Cloud storage of geotagged maps

ABSTRACT

A head mounted device provides an immersive virtual or augmented reality experience for viewing data and enabling collaboration among multiple users. Rendering images in a virtual or augmented reality system may include generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device, generating a three dimensional map of the scene based on the generated data, adding geographical identification metadata to the three dimensional map of the scene, storing the geographical identification metadata and three dimensional map in a memory, and transmitting at least a portion of the geographical identification metadata and three dimensional map to a second body mounted sensor device.

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This patent application claims the benefit of priority to U.S.Provisional Patent Application Ser. No. 61/468,937 entitled “Systems andMethods for Gesture Driven Interaction for Digitally Augmented PhysicalSpaces” filed on Mar. 29, 2011, the entire contents of which are herebyincorporated by reference for all purposes.

This patent application is also related to U.S. patent application Ser.No. ______ entitled “Modular Mobile Connected Pico Projectors For ALocal Multi-User Collaboration” filed on ______, U.S. patent applicationSer. No. ______ entitled “Anchoring Virtual Images to Real WorldSurfaces In Augmented Reality Systems” filed on ______, U.S. patentapplication Ser. No. ______ entitled “Selective Hand Occlusion OverVirtual Projections onto Physical Surfaces Using Skeletal Tracking”filed on ______, and U.S. patent application Ser. No. ______ entitled“System For The Rendering Of Shared Digital Interfaces Relative To EachUser's Point Of View” filed on ______.

FIELD OF THE INVENTION

The present application relates to an augmented or virtual realitysystem using a head mounted display, or other mobile devices such assmartphones or tablets, that can place a virtual object or interface ona selected physical surface so that a single user or multiple users cancollaborate to, view and interact with the virtual object on thephysical surface.

BACKGROUND

Increasingly, people are collaborating around the globe on a variety ofmedia, including presentations, documents, videos, graphs andphotographs. Generally, large flat panel displays in a conference roomare an excellent source to view media including Microsoft® PowerPoint®presentations. Additionally, some furniture may include touch screeninput devices so users can view video directly on the surface of a tabletop. Often such large items are very expensive and provide limitedsupport for collaboration between users in remote locations. Individualswould benefit from being able to break free from these restrictions andgain a big screen virtual or augmented reality experience that is sharedbetween both co-located users and remote users.

SUMMARY OF THE INVENTION

The various embodiments include methods of storing spatial data for usein an augmented reality system, including generating data regardinglocations of surfaces and objects in a scene based on images and spatialdata gathered from a first body mounted sensor device, generating athree dimensional map of the scene based on the generated data, addinggeographical identification metadata to the three dimensional map of thescene, storing the geographical identification metadata and threedimensional map in a memory, and transmitting at least a portion of thegeographical identification metadata and three dimensional map to asecond body mounted sensor device. In an embodiment, generating dataregarding locations of surfaces and objects in a scene based on imagesand spatial data gathered from a first body mounted sensor device mayinclude capturing a scene image of the scene with a body mounted camera,capturing spatial data of objects within the scene with a body mountedsensor array, recognizing objects within the captured scene image, anddetermining distances from the first body mounted sensor device to theobjects recognized within the scene image, and adding geographicalidentification metadata to the three dimensional map of the scene mayinclude associating the determined distances with the objects recognizedwithin the scene image.

In an embodiment, the method may include determining a location of thesecond body mounted sensor device, in which transmitting at least aportion of the geographical identification metadata and threedimensional map to the second body mounted sensor device may includetransmitting a portion of the geographical identification metadata andthree dimensional map relevant to the location of the second bodymounted sensor device. In a further embodiment, the method may includereceiving the transmitted portion of the geographical identificationmetadata and three dimensional map in the second body mounted sensordevice, and generating a visual image on a display on the second bodymounted sensor device based in part on the received portion of thegeographical identification metadata and three dimensional map.

In a further embodiment, the method may include determining a locationof the second body mounted sensor device, generating data regardinglocations of surfaces and objects in the scene based on images andspatial data gathered from the second body mounted sensor device, andrefining the geographical identification metadata and the threedimensional map based upon the data generated by the second body mountedsensor device. In a further embodiment, adding geographicalidentification metadata to the three dimensional map of the scene mayinclude tagging the three dimensional map with a geotag. In a furtherembodiment, generating data regarding locations of surfaces and objectsin a scene based on images and spatial data gathered from a first bodymounted sensor device, generating a three dimensional map of the scenebased on the generated data, and adding geographical identificationmetadata to the three dimensional map of the scene may be accomplishedin a processor within the first body mounted sensor device.

In a further embodiment, the method may include relaying images andspatial data gathered by the first body mounted sensor device to aprocessor in communication with the first body mounted sensor device, inwhich generating data regarding locations of surfaces and objects in ascene based on images and spatial data gathered from a first bodymounted sensor device, generating a three dimensional map of the scenebased on the generated data, and adding geographical identificationmetadata to the three dimensional map of the scene may be accomplishedin the processor in communication with the first body mounted sensordevice. In a further embodiment, the method may include transmittingimages and spatial data gathered by the first body mounted sensor deviceto a server via a network, in which generating data regarding locationsof surfaces and objects in a scene based on images and spatial datagathered from a first body mounted sensor device, generating a threedimensional map of the scene based on the generated data, and addinggeographical identification metadata to the three dimensional map of thescene may be accomplished in the server based on the image and spatialdata received from the first body mounted sensor device.

In a further embodiment, the method may include determining a locationof the second body mounted sensor device, and transmitting a requestfrom the second body mounted sensor device to the server for a portionof the geographical identification metadata and three dimensional map,the request including the determined location of the second body mountedsensor device, in which transmitting at least a portion of thegeographical identification metadata and three dimensional map to thesecond body mounted sensor device may include transmitting from theserver to the second body mounted sensor device a portion of thegeographical identification metadata and three dimensional map that isrelevant to the location of the second body mounted sensor device.

In a further embodiment, generating a three dimensional map of the scenemay include developing a mathematical representation ofthree-dimensional surfaces within the scene, and writing thegeographical identification metadata into the mathematicalrepresentation. In a further embodiment, storing the geographicalidentification metadata and the three dimensional map to a memory mayinclude storing the mathematical representation on a server accessibleby the first and second body mounted sensor devices. In a furtherembodiment, generating data regarding locations of surfaces and objectsin a scene based on images and spatial data gathered from a first bodymounted sensor device may include determining a location of the firstbody mounted sensor device, capturing a scene image of the scene with abody mounted camera, capturing spatial data of objects within the scenewith a body mounted sensor array, recognizing objects within thecaptured scene image, and determining locations of the objectsrecognized within the scene image, and adding geographicalidentification metadata to the three dimensional map of the scene mayinclude associating the determined locations with the objects recognizedwithin the scene image. In a further embodiment, determining locationsof the objects recognized within the image may include determininglocations of the objects with respect to the determined location of thefirst body mounted sensor device. In a further embodiment, determininglocations of the objects recognized within the image may includedetermining locations of the objects within a coordinate system used indetermining the location of the first body mounted sensor device.

Further embodiments include a system that may include a first head orbody mounted device that may include a first head or body mountedcamera, a first head or body mounted display, a first transceiver, and afirst head or body mounted device processor coupled to the first camera,first display and first transceiver; and a second head or body mounteddevice comprising a second head or body mounted camera, a second head orbody mounted display, a second transceiver, and a second head or bodymounted device processor coupled to the second camera, second displayand second transceiver. In further embodiments, the first and secondhead or body mounted device may be configured with processor-executableinstructions to perform various operations corresponding to the methodoperations discussed above. Even further embodiments may include aserver that may include a memory and a server processor coupled to thememory, the server processor being configured with server-executableinstructions to perform operations corresponding to the various methodsdiscussed above and/or for transmitting information to the first orsecond head or body mounted devices via a network.

Further embodiments include a computing device that may include aprocessor configured with processor-executable instructions to performvarious operations corresponding to the methods discussed above. Furtherembodiments include a computing device that may include various meansfor performing functions corresponding to the method operationsdiscussed above.

Further embodiments include a non-transitory processor-readable storagemedium having stored thereon processor-executable instructionsconfigured to cause a processor to perform various operationscorresponding to the methods discussed above.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitutepart of this specification, illustrate exemplary embodiments of theinvention, and together with the general description given above and thedetailed description given below, serve to explain the features of theinvention.

FIG. 1 is an illustration of a head mounted device with a virtual objectanchored on a physical surface suitable for use with variousembodiments.

FIG. 2 is an illustration of two users standing at a table side by sidefrom one another and wearing two head mounted devices for interactingwith a virtual object.

FIG. 3 is an illustration of two users sitting at a table across fromone another and wearing two head mounted devices for interacting with avirtual object and with the virtual object being correctly oriented foreach user.

FIG. 4 is an illustration of two users at two different remote locationswearing two head mounted devices and collaborating and studying the samevirtual object, which is correctly oriented for each user.

FIG. 5A is a high level component block diagram of a number ofcomponents of a head mounted device to interact with and to generate avirtual object.

FIG. 5B is a high level component block diagram of a system that enablesusers wearing a head mounted device to communicate with a serveraccording to another embodiment.

FIG. 6 is a process flow diagram of an embodiment method for using ahead mounted device to anchor a virtual object on an anchor surface inwhich the virtual object remains fixed to the anchor surface on the headmounted display.

FIG. 7A is a process flow diagram of an embodiment method for anchoringa virtual object on a physical surface viewable in a display of a headmounted device.

FIG. 7B is a process flow diagram of an embodiment method for moving avirtual object between at least two different anchor points or surfacesby an input command.

FIG. 8A is a process flow diagram of an embodiment method for a headmounted device to project a virtual object so it appears anchored on aphysical surface and for projecting a second virtual object on a secondhead mounted device so that it appears to a second user to be anchoredon a different physical surface.

FIG. 8B is a process flow diagram of an embodiment method of rendering ashared virtual object that is oriented correctly for each user's pointof view.

FIGS. 9A and 9B are process flow diagrams of an embodiment method foranchoring a virtual object on a physical surface and for using a seconddevice to anchor a second virtual object on a different physical surfaceusing a coordinate map.

FIG. 10 is a process flow diagram of an embodiment method fordiscovering other head mounted devices to link and share data.

FIG. 11 is a process flow diagram of an embodiment method forrecognizing both gestures and audio from a user as a confirmed inputcommand for controlling the head mounted device.

FIG. 12 is a process flow diagram of an embodiment method for using ahead mounted device that captures images of a user to recognize gesturesas input commands.

FIG. 13 is a process flow diagram of an embodiment method forcontrolling the rendering of certain objects within an image.

FIG. 14 is a process flow diagram of an embodiment method for using ahead mounted device to capture images associated with a user and todetect the user's and other individual's features, and when the otherindividual's features are detected, to superimpose the virtual objectover the other individual's features so as to not occlude the virtualobject.

FIG. 15 is a top view of a head mounted display output showing a virtualobject with a user's hands and with other individual's hands occludingthe virtual object and with the virtual object displayed superimposed onthe other individual's hands so they seem transparent.

FIG. 16 is a side view of a head mounted device communicating with amobile communication device via a high speed wireless data link sosoftware is executed on the mobile communication device while thesensors capture data at the head mounted device.

FIG. 17 is a process flow diagram of an embodiment method for using ahead mounted device to wirelessly link to the mobile communicationdevice to exchange data.

FIG. 18 is a process flow diagram of an embodiment method for using ahead mounted device to scan an area, calculate parameters of an area,and upload the calculated parameters of the area to a shared memory foraccess by another device.

FIG. 19 is a process flow diagram of an embodiment method for using ahead mounted device to generate a three-dimensional model of an area,tag the data with location data, and upload the model to share the datawith other devices.

FIG. 20 is a process flow diagram of an embodiment method for using ahead mounted device to execute an application on a server and share thedata with other devices.

FIG. 21 is a process flow diagram of an embodiment method for using ahead mounted device to discover other head mounted devices by adirectory to share data with other devices and for collaboration.

FIGS. 22-23 illustrate two head mounted devices accessing a sharedservice or a shared memory to exchange scanned data and to collaboratewith one another.

FIG. 24 is a process flow diagram of an embodiment method for using ahead mounted device to scan an area, render a virtual object, tagcalculated parameters with location data, and upload the taggedparameters to a shared memory for the benefit of other head mounteddevices.

FIG. 25A is a high level illustration of several mobile devicescommunicating with a cloud server to provide shared data to the cloudserver and to allow new devices to scan and map an area utilizing theshared data.

FIG. 25B is a process flow diagram of a method of storing tagged andscanned parameters in a cloud server for use by a second head mounteddevice to render a virtual object.

FIGS. 26-28B are process flow diagrams of embodiment methods for using ahead mounted device to render a virtual object on a display and tosuperimpose the virtual object over a body part on the display.

FIG. 29 is a process flow diagram of an embodiment method for using ahead mounted device and a cloud server to share scanned and calculateddata parameters with several other head mounted devices.

FIG. 30 is a process flow diagram of an embodiment method for using twodifferent head mounted devices and a server to share data andcollaborate using two different software applications using a virtualobject input device whereby conflicts are resolved between the twodifferent software applications.

FIG. 31 is a high level block diagram of several users sharing data andcollaborating using different software applications.

FIG. 32 is a process flow diagram of an embodiment method for using twodifferent computing devices to share data and collaborate using twodifferent word processing applications.

FIGS. 33-34 are process flow diagrams of embodiment methods for using ahead mounted device to recognize both gestures and audio commands, andto process the commands as confirmed input commands when a first commandand a second command are detected in succession.

FIGS. 35-36 are process flow diagrams of embodiment methods for using ahead mounted device to recognize gesture commands to render a “virtualobject input device” on the generated virtual object so the user maycontrol the virtual object using the virtual input device.

FIG. 37 is a high level block diagram of an embodiment using a gesturetracking camera and/or an RGB and distance measuring camera (or cameras)to provide the “virtual object input device” which is a virtual pointerdevice that is controlled by a user's finger to select icons on thevirtual object.

FIGS. 38-39 show high level block diagrams of a peer to peer basedsystem and a central cloud configuration based system for using a headmounted device and a server to share data and enable user collaboration.

FIG. 40A is a high level block diagram of an embodiment system using agesture tracking camera and a camera to provide the “virtual objectinput device” that forms a virtual pointer device where the virtualobject may be rendered large on the head mounted display.

FIGS. 40B-40C show another embodiment of rendering a virtual objectusing a modular pico projector that may detect gestures and output aprojection for a user.

FIG. 40D shows a process flow diagram of an embodiment method ofprojecting an image with a pico projector in a convenient location for auser by detecting one or more gestures.

FIG. 41 is a component block diagram of a mobile computing devicesuitable for use with the various embodiments.

FIG. 42 is a component block diagram of a server computing devicesuitable for use with the various embodiments.

DETAILED DESCRIPTION

The various embodiments will be described in detail with reference tothe accompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.References made to particular examples and implementations are forillustrative purposes, and are not intended to limit the scope of theinvention or the claims.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any implementation described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other implementations.

As used herein, the terms “mobile device” and “handheld device” refer toany one of cellular telephones, smartphones, tablet computers, personaldata assistants (PDA's), wireless electronic mail receivers, multimediaInternet enabled cellular telephones, Global Positioning System (GPS)receivers, wireless gaming controllers, netbooks, and similar personalelectronic devices that include a programmable processor and memory, areconfigured to communicate with a wireless communication network, andhave a web browser.

As used herein, the terms “computer,” “personal computer” and “computingdevice” refer to any programmable computer system that is known or thatwill be developed in the future. In a preferred embodiment a computerwill be coupled to a network such as described herein. A computer systemmay be configured with processor-executable software instructions toperform the processes described herein.

As used herein, the term “component,” “module,” and “system,” isintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on a server and the server maybe a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers.

As used herein, the term “head mounted device” (HMD) refers to a devicethat captures distance sensor data and has a display capability linkedto a mobile processor, which may be a separate device relative to thehead mounted device and as shown in FIG. 16. In an embodiment, the headmounted device 10 may be an accessory for a mobile device CPU (e.g., theprocessor of a cell phone, tablet computer, smartphone, etc.) with themain processing of the head mounted devices control system 1610 (“VIRTcontrol system” in the figures) being performed on the processor ofmobile device 1605. In another embodiment, the head mounted device maycomprise a processor, a memory, a display and a camera. In anembodiment, head mounted device 10 may be a mobile device (e.g.,smartphone, etc.) that includes one or more sensors (e.g., a depthsensor, camera, etc.) for scanning or collecting information from anenvironment (e.g., room, etc.) and circuitry for transmitting thecollected information to another device (e.g., server, second mobiledevice, etc.).

In another embodiment, the head mounted device may include a wirelessinterface for connecting with the Internet, a local wireless network, oranother computing device. In another embodiment, a pico-projector may beassociated in the head mounted device to enable projection of imagesonto surfaces. The head mounted device is preferably lightweight andconstructed to avoid use of heavy components, which could cause thedevice to be uncomfortable to wear. The head mounted device may also beoperable to receive audio/gestural inputs from a user. Such gestural oraudio inputs may be spoken voice commands or a recognized user gesture,which when recognized by a computing device may cause that device toexecute a corresponding command.

The term “peer-to-peer network” is intended to encompass any form ofpeer-to-peer network technology known or that may be developed in thefuture. While some embodiments refer to peer-to-peer networks that makeuse of hypertext transfer protocol (HTTP) messaging, such references areintended merely to serve as examples, and not to limit the scope of theclaims to any particular networking technology.

For ease of reference, examples illustrating the functioning of variousembodiments may refer to networks as first and second networks or users,or to networks and users A and B. For example, a first user may bereferred to as “user A” and a second user may be referred to as “userB.” Such references in the figures and the descriptions are arbitraryand used to simplify network and user references, and thus are notintended to limit the scope of the claims to just two networks or justtwo users, or to limit particular users to particular networks, as thevarious embodiments support any number of networks and users.

Various embodiments will be presented in terms of systems that mayinclude a number of components, and modules. It is to be understood andappreciated that the various systems may include additional components,modules, etc., and may not include all of the components, modules, etc.discussed in connection with the figures. A combination of theseapproaches may also be used.

Turning to FIG. 1, the various embodiments enable a head mounted device10 to render a virtual object 14 displayed on an anchored surface 16 inorder to provide an augmented reality experience that can facilitateinteractions with a computing device and collaborations with otherusers. The various embodiments may include capturing an image with acamera that is head mounted or body mounted (e.g., on the neck, shoulderor arms of a user, or hand held). For ease of reference, the term “bodymounted” as used herein and in the claims encompasses head mounted. Inan embodiment, this camera may be a head or body mounted stereo camera,which can generate image data that a processor can analyze to estimatedistances to objects in the image through trigonometric analysis ofstereo images. Alternatively or in addition, the head mounted device mayinclude one or more distance measuring sensors (e.g., a laser or sonicrange finder) that can measure distances to various surfaces within theimage. As discussed in more detail below, in the various embodiments avariety of different types of distance measuring sensors and algorithmsmay be used an imaged scene to measure for measuring distances toobjects within a scene viewed by a user. Also, more than one sensor andtype of sensor may be used in a head mounted device. Therefore, for easeof description and consistency, the various assemblages and types ofdistance measuring sensors that may be included on a head mounted deviceare referred to herein collectively or individually as “distancesensors.”

Further, the head mounted device 10 may include orientation sensors,such as accelerometers, gyroscopes, magnetic sensors, optical sensors,mechanical or electronic level sensors, and inertial sensors which aloneor in combination can provide data to the device's processor regardingthe up/down/level orientation of the device (e.g., by sensing thegravity force orientation) and thus the user's head position/orientation(and from that viewing perspective). Further, the head mounted devicemay include rotational orientation sensors, such as an electroniccompass and accelerometers, that can provide data to the device'sprocessor regarding left/right orientation and movement. Collectively,sensors (including accelerometers, gyroscopes, magnetic sensors, opticalsensors, mechanical or electronic level sensors, inertial sensors, andelectronic compasses) configured to provide data regarding the up/downand rotational orientation of the head mounted device (and thus theuser's viewing perspective) are referred to herein as “orientationsensors.”

The system may be configured to recognize user inputs, which may be madethrough gestures that may be imaged by the camera, and identify surfacesor locations for positioning virtual objects within the imaged scene. Adistance to the recognized object within the image may be determinedfrom data gathered from a stereo image and/or a distance sensor. Thehead mounted device 10 may provide image and distance sensor data to andreceive display information from a mobile processor which may beseparate from the head mounted device, such as in a smartphone or othermobile device, as discussed in more detail below with reference to FIG.16. This process of orienting the head mounted device with respect tothe surroundings, and determining a distance to and orientation ofvarious surfaces may be accomplished by each device being worn by anumber of users so that a displayed virtual object is seen by each userfrom the perspective appropriate for each user. Further, this processmay be accomplished continuously so that each head mounted device cantriangulate its own position and angle of view as it moves in space byconstantly referencing the shifts in the topography of scannedthree-dimensional (3D) environment, thus performing SimultaneousLocation and Mapping (SLAM) operations.

In an embodiment, the head mounted device 10 may receive a first userinput indicating a first anchor surface 16 in a captured image. Thisinput may be any of a variety of user inputs, such as in the form of abutton press, a recognizable gesture performed in view of thehead-mounted camera, a focused gaze by the user recognized as an imagethat remains relatively still for a predetermined period of time, orother recognizable input. The first anchor surface 16 may correspond toa first surface located in the image obtained by the camera of the headmounted device 10. A processor coupled to or within the head mounteddevice 10 may generate a virtual object 14 and calculatedisplay-relevant parameters, including distance and orientation withrespect to the head mounted or body mounted camera that correspond to adisplay location of the virtual object 14. When the user elects todisplay the virtual object 14 anchored to a designated anchor surface,the processor coupled to or within the head mounted device 10 may renderthe virtual object 14 on a display (or through a projector) so that thevirtual object appears to be on the first anchor surface 16 when viewedthrough the display. The virtual object 14 may be any virtual object 14,including, for example, text, graphics, images and 3D shapes. Whenpresented in this manner, the projection of virtual objects positionedat/on designated locations within the surrounding environment can createthe experience of virtual reality and enable user interactions with thevirtual object. The various embodiments enable natural interactions withvirtual objects and digital assets (documents, pictures, videos, etc.).Such natural interactions with virtual objects and digital assets mayinclude gesture controls, touch manipulations, highlighting of touchedportions of the virtual object, etc. Recognizable gestures may be storedor organized in the form of a gesture dictionary accessible by headmounted devices. Such a gesture dictionary may store movement data orpatterns for recognizing gestures that may include pokes, pats, taps,pushes, guiding, flicks, turning, rotating, grabbing and pulling, twohands with palms open for panning images, drawing (e.g., fingerpainting), forming shapes with fingers (e.g., an “OK” sign), and swipes,all of which may be accomplished on, in close proximity to, oraddressing the direction of (in relation to the user) the apparentlocation of a virtual object in a generated display. In this manner, thevarious embodiments enable natural interactions with virtual objects,which may include gesture controls or other suitable control inputmethods.

The various embodiments may be implemented using different types of headmounted displays. Some head mounted displays may be semitransparentenabling the user to view the scene beyond the display, with projectedimages appearing superimposed upon the background scene. In otherembodiments, the head mounted display completely blocks the user's viewof the room. In this embodiment, the head mounted display provides avirtual or augmented reality experience. In order to see the objects inthe room, images obtained by a camera or cameras worn by the user may beused to project a view of the room onto the head mounted display. Inthis embodiment, virtual objects may be added to the images projected onhead mounted display, thereby appearing as real as the actual imagesobtained by the cameras.

In embodiments in which the head mounted display is nontransparent (i.e.fully occluded), the head mounted device 10 may display images of thescene in front of the user with virtual objects at 14 included in thedisplay so that they appear anchored to the selected anchor surface 16.In this embodiment, the imaging sensors and head mounted device projectonto the nontransparent display an image of the scene in front of theuser. In order for the user to navigate to that scene, the user viewsthe images obtained from the head mounted cameras on the head mounteddisplay. This may include rendering into the display images of theuser's hands, particularly when the user is moving his or her hands aspart of a control gesture. If the virtual object 14 is anchored to aparticular surface, the virtual object will not appear in the displaywhen the user is looking away from the anchor surface. Since the headmounted display presents images generated by the head mounted device,including images obtained from body-mounted cameras, the system maydisplay portions of the scene so virtual objects appear in front of thebackground. Thus, in this implementation, if a virtual image ispresented on the head mounted display in front of the user's hands, theuser's hands would not be visible to the user. As a result the virtualobject may appear to be solid and nontransparent. This embodimentprovides a virtual or augmented reality experience that includesdisplaying real images obtained by user-worn cameras combined withvirtual objects presented on the same display.

In embodiments in which the head mounted display is semitransparent(i.e., user can see the background scene through the display), the headmounted device 10 may display the generated virtual object so thevirtual object appears anchored to the first anchor surface 16 as seenby the user through the display. In this manner, the user may view ascene with a partially transparent head mounted display where real worldobjects, like a desk, a table and walls, are partially visible throughthe head mounted display which also places virtual objects within thevisible scene. Visible virtual objects 14 may be anchored to orconnected to real world objects 16. This embodiment provides anaugmented reality experience in which the display is see-through orvideo see-through, allowing the user to see the real world through thedisplay with virtual objects appearing to be fixed in real locations oron real surfaces.

For example, a virtual object 14 may resemble a flat screen televisionand may be connected to or “anchor” to a real world object or a realworld wall 16, as illustrated in FIG. 1. As the user moves his or herhead, the virtual object 14 normally would move on the display 10 withthe user's field of vision. However, in this embodiment the virtualobject 14 appears to remain on the anchor surface 16, similar to how areal world flat panel display would remain on a surface if the userturned his or her head.

In an embodiment, users may move the anchor surface 16 in a mannersimilar to how the user selected the first anchor surface. The user maydiscover that for a certain task that a different anchor surface will bepreferable. The head mounted device 10 may receive a second input(gesture, audio, from an input device, etc.) indicating a new or asecond anchor surface 16 within the image that is different from thefirst anchor surface 16. The second anchor surface 16 may correspond toa second different surface located in the image. Further, the first andsecond anchor surfaces may not be adjacent and the first surface may notbe in view of the head mounted device cameras when thesecond/alternative surface is designated. For example, one surface mightbe a desktop 16 as shown in FIG. 2, while another surface may be ahorizontal wall 16 or a ceiling as shown in FIG. 1. For example, a firstuser may select a first anchor surface 16 for personal usage and thenselect a second anchor surface 16 for a second user in a differentgeographic location. In an embodiment, the user inputs may be voiceinputs, inputs provided using a tangible input device (keyboard ormouse), detected gestures, or may be provided by different users. Aprocessor within or coupled to the head mounted device 10 may calculateparameters, including distance and orientation with respect to the headmounted or body mounted camera that corresponds to the second anchorsurface 16. The processor within or coupled to the head mounted device10 may then display the generated virtual object 14 so the virtualobject appears to the user to be anchored to the selected second anchorsurface 16. In another embodiment, instead of or in addition to a headmounted device 10, a pico projector may be used to project a virtualobject 14 onto the selected anchor surface 16. The pico projector may bea separate modular device, and or may be included within the headmounted device 10.

FIG. 1 illustrates a user using a head mounted device 10. The headmounted device 10 may enable the user to designate nearby surfaces, suchas a wall 12, as a virtual “flat panel screen” 16. The user may identifya nearby surface 16 using an input command. When multiple users areequipped with head mounted devices 10 connected to a common network ordatabase, some or all of those users may see the same virtual object 14at the same time, which enables collaboration and group discussionsregarding the virtual object as if it were a real world object.

The processor driving the display on the head mounted device 10 maycontinuously update the displayed image of the generated virtual object14 so that the virtual object 14 appears anchored to the anchor surface16 as the user turns his/her head and moves with respect to the selectedanchor surface 16. Thus, as the user moves and walks around the room,the virtual object 14 may appear to remain fixed on the physical surface16. As part of maintaining this impression, the processor may modify theappearance of the virtual object 14 so that its shape, size andorientation match the user's viewing perspective of the anchor surface.Adjusting the shape, size and orientation of virtual objects may help toprovide an immersive virtual or augmented reality experience. Unless theuser enters a predefined specific input to move the virtual object 14 toanother anchor surface, the virtual object 14 may remain static andseemingly adhered to the anchor surface. This may be beneficial forcollaboration among a group of users, because the virtual object 14 maybe a collaborative focal point for a conversation regardless of thelocation and viewing perspective of each user. For example, a first usermay read media on the anchor surface or watch a presentation given by asecond user. This will give the first user the appearance that thevirtual object 14 is a real world or physical object similar to a flatpanel screen, except that the first user may change the anchor surface16 to a different surface or object in the room without disruption.

In a non-limiting example, the virtual object 14 may be a display outputof a software application intended for collaboration. For example, thevirtual object 14 may include text, media, television channels, movies,document word processing applications, email, video, telephone calls,social network postings, RSS feeds, etc. Such objects may be presentedon the display to appear fixed to or on an anchor surface, similar tohow the objects might appear on a conventional flat panel display,except that any surface may serve as the display. Additionally, the sizeof the virtual object anchored to a selected surface may be controlledby the user to enable the displayed object to be made larger or smaller.In another embodiment, the virtual object 14 may have a differentappearance, such as being rendered on non-planar and non-rectangularsurfaces, or rendered with a spherical or a seemingly three-dimensionalshape. The presentation of the virtual object 14 on the head mounteddisplay allows users to view and interact with the object as if it werea part of the physical environment. In an embodiment, the virtual object14 may look different to each user.

In an embodiment, the virtual object 14 may be a three-dimensional modelthat is not anchored to a physical surface 16. In this embodiment, thevirtual object 14 may be a three-dimensional model that appears floatingin space. In this embodiment, the user may view the displayed virtualobject 14 in free space with the virtual object 14 anchored to aselected three-dimensional position in free space which may be relativeto walls, floors, furniture etc. In this embodiment, when the virtualobject 14 is anchored to a particular location in space, the object willnot appear to move as the user moves his/her head and moves around theanchor location, and thus around the virtual object. For example, thevirtual object 14 may be presented on the partially transparent displayso that it appears anchored in free space at a predetermined 3-Dcoordinate. In this embodiment, the display may display a prompt for theuser to input a place or coordinates at which the system should displaythe virtual object 14 as free floating. The processor may recognize aninput command via a detected gesture (e.g., a finger pointing to a pointin space) or may recognize an audio command to place the virtual object14 as free floating in free space. The processor may display the virtualobject 14 anchored to the indicated spatial point, or may connect thevirtual object 14 to a position of the head mounted display so thevirtual object 14 has the illusion of floating in space. As the usermoves relative to the virtual object 14 in free space, the perspective,size, and orientation of the virtual object 14 may also changeaccordingly. For example, the virtual object 14 may be anchored to emptyspace located adjacent to the wall or outside floating above the groundand outside of a window. In this manner, users may inspect all sides ofthe virtual object 14 by walking around the free floating virtual object14 in the room.

In the various embodiments, the input to anchor the virtual object 14 ona selected anchor surface or point in space may be provided in a numberof different ways. For example, the camera associated with the headmounted device 10, or a second camera operatively connected to the headmounted device 10 may also track movements of the user. The processormay receive the images and may apply a detection algorithm to thetracked movements to detect a predetermined gesture made by the user.For example, the user may point to a surface, which the system mayinterpret as a specific input to the processor.

For example, the user may indicate with a gesture that a particular realworld surface in the room is intended to be an anchor surface. Theprocessor may execute a command corresponding to the predeterminedgesture when the predetermined gesture is detected. For example, theprocessor may detect that a user is pointing to a particular surface.The processor may detect the surface and determine an angle of thesurface with respect to the line of sight of the user, and anchor thevirtual object 14 on the particular surface where the user pointed withan orientation and perspective consistent with the determined angle ofthe surface.

For example, the user may point to a desktop surface. The processor maybe tracking the user's movements and may apply a detection algorithm tothe tracked images. When the system images the user's arm outstretchedwith the index finger pointing and the remaining digits forming a fist,the processor may recognize such an image as user gesture and apply acorresponding command to anchor the virtual object 14 on the desktopwhere the user is pointing. For example, if the end of a user's fingercontacts a surface to identify it as the anchor surface, the headmounted device may display the virtual object 14 as anchored to thatsurface.

In another example, a user may desire to move the virtual object 14 to“place” it on a different anchor surface 16. To accomplish this, theuser may provide or execute a second predetermined gesture that aprocessor coupled to the head mounted device 10 may recognize. Thesecond predetermined gesture may be the same as that used to designatethe first anchor surface, such as a general command to designate asurface within the captured image as an anchor surface 16, or may be adifferent gesture. Many different gestures can be made and are withinthe scope of the present disclosure. For example, the detectedpredetermined gesture may be an “OK” sign, a fist, an open hand,pointing with one finger, pointing with two fingers, pointing with threeor four fingers, an outstretched hand, a hand rotation, a wave, amovement of one or more fingers, a movement of a body part (e.g.,movement of a foot), or other arbitrary movements.

Also, designation of an anchor surface 16 and other inputs or commandsmay be provided verbally with the input received by a microphone on orcoupled to the head mounted device 10 or coupled to the processor inturn coupled to the head mounted device. The processor within or coupledto the head mounted device may receive audio signals from themicrophone, and may process such signals using speech recognitionprocesses. The processor may compare received audio signals to audiopatterns of one or more commands stored in a memory in order torecognize a spoken command. For example, the processor may monitor audioinputs for a few predetermined command words. In this manner, the usermay speak with colleagues and other networked users, such as to discussvirtual objects, without saying a command keyword. The processor mayapply a detection algorithm to the received audio so that it onlyresponds to particular predefined audio commands, or commands proceededby a predefined attention command (e.g., “computer” or “execute” etc.).For example, the user may say “anchor image on desk” to cause theprocessor to anchor the virtual object to a desk within the field ofview of the head mounted device camera. The processor within or coupledto the head mounted device 10 may recognize these spoken words as acommand input, and implement corresponding actions to anchor the virtualobject 14 “on” the desk or anchor surface 16.

In another embodiment, the processor may be programmed to detect commandinputs, but then wait for a confirmation prior to implementing anycommand. This confirmation may be advantageous to avoid inadvertentlymoving a virtual object 14 by an unintentional gesture or by aninadvertent audio command during a group collaboration regarding theobject. In this embodiment, the virtual object 14 will not be placed,moved or anchored to a surface in response to a command unless aconfirmation or confirming command is received. For example, theprocessor may process images from a camera to identify gesture commandsand/or audio signals received from a microphone to identify spokencommands, such as a single word or a sentence. When a gesture or spokencommand is recognized, the processor may prompt the user to confirm thecommand and await confirmation input, which may be in the form of abutton press, a spoken word, a physical gesture, or combinationsthereof. When the processor recognizes the appropriate confirmation orconfirming command, it may execute the command associated with the firstrecognized gesture or audible command. If the confirmation command isnot received, the processor may ignore the detected gesture or audiblecommand.

In another embodiment, a mobile computing device, such as a smartphonewith camera may operate as a ‘magic lens’ for users that are not wearinga head mounted display. In this embodiment, a user with a computingdevice connected to a system via a wireless data link may view thevirtual objects on the device's display. This allows those users withouta head mounted device to view the virtual objects and structures in theimage, such as the anchor surface, in a manner similar to the way itappears on head mounted displays. For example, a first user may view avirtual object 14 connected to a wall on a head mounted display whichcommunicates the image data to a second user's smartphone or PDA displayso the second user may view the virtual object 14 as it appears anchoredon a wall or other surface. In another embodiment, modular picoprojectors may be linked to a mobile device and used to project imagesof virtual objects.

In various embodiments, the head mounted devices 10 include wirelesscommunication capabilities which enable exchanging of data between twoor more devices to enable two or more users to interact with andcollaborate on documents, files and virtual objects. In this manner, thehead mounted devices 10 can facilitate collaboration by multiple usersover virtual objects, such as providing a virtual or augmented realityexperience enabling users to interact with and view three-dimensionalvirtual objects from various perspectives. To enable such acollaboration experience, the head mounted devices 10 may transmit toeach other three-dimensional virtual object models and/or data sets forrendering on their respective displays. The head mounted devices 10 mayalso receive data files, such as coordinate maps and three-dimensionalvirtual object models and/or data sets from other computing devices,such as network servers. Also, image data generated by one or more ofthe head mounted devices involved in a collaboration may be transmittedto and displayed on head mounted devices or other mobile devices such assmartphones or tablet computers used by non-participants, enablingothers to view the collaboration in a virtual or augmented realityexperience. Others viewing a collaboration session wearing head mounteddevices or using another mobile device such as a smartphone or tabletmay not only see the virtual objects and user interactions with them,but have limited interaction capabilities with the virtual augmentationsseen by one of the head mounted device users. This limited interactionmay include touching the augmentation to cause an effect, defining aninteractive area or anchor point on the physical surface (effectivelyadding a new augmentation to the shared experience), and interactingwith the shared mixed reality scene via gestural and/or audio inputs.Any of such changes in the virtual experience may be reflected in thehead mounted displays worn by the other users.

FIG. 2 illustrates two users using first and second head mounted devices10 a, 10 b to view virtual objects anchored to a surface, such as a desktable top 16. In the illustrated example, the anchor surface 16corresponds to the desk top surface 16, which is located within theimage displayed by the first head mounted device 10 a. The head mounteddevice 10 a may generate a virtual object 14 a, which is shown as avirtual screen in this illustration, and display the virtual object 14 aas if it were connected to the desk top surface 16. If the first userlooks to the left, to the right or rearward, the virtual object 14 awill not travel within the field of view of the head mounted device 10a, and instead will remain displayed on the desktop surface 16 anchoredin place. For example, the virtual object 14 a will no longer appearwithin the head mounted device screen if the first user looks away fromthe anchor surface 16. Also, the virtual object 14 a may be presented tothe first user in the orientation, shape and size, as it would appear ifthe anchor surface 16 were a flat panel display. Thus, the virtualobject 14 a may be displayed with a shape and size resulting fromprojecting the object's shape onto the anchor surface from the user'sperspective. To accomplish this, the processor within or coupled to thehead mounted device 10 a may calculate shape parameters includingdistance and orientation with respect to the head mounted or bodymounted camera 10 a and distance sensors in order the generate a virtualobject image at or on the anchor surface. Such calculations mayimplement well known vector projection algorithms to yield the imagethat is presented in the head mounted display. By applying vectorprocessing of the image consistent with normal experience (e.g., normalEuclidean geometric projections) and presenting the resulting virtualobject 14 b on a semitransparent display that also enables the user tosee objects beyond the display, the result can fool the senses of theuser so the virtual objects appear to behave as real objects.

For collaborative purposes, a second user may wear a second head mounteddevice 10 b to view the same virtual object within the same physicalspace. The processor within or coupled to the second head mounted device10 b may render the virtual object on a user-selected anchor surface 16.The second head mounted device 10 b may display the virtual object 14 bon the same anchor surface or position as designated for the first headmounted device 10 a. The second user may also designate a differentposition or anchor surface for rendering the virtual object 14 b as seenthrough the second head mounted device 10 b. In order to enable thesecond head mounted device 10 b to properly render the virtual object 14b on the anchor surface from the second user's perspective, the datatransmitted from the first head mounted device to the second headmounted device may include the shape or object data. This data mayenable the second head mounted device processor to render a displayedimage of the virtual object corresponding to the second user's viewingperspective. The virtual object data may be in the form of a geometricmodel, coordinates and fill data, or similar rendering data that may beused in a three-dimensional object rendering module implemented in aprocessor within the second head mounted device 10 b.

In some implementations or situations, the second head mounted device 10b may receive image data from the first head mounted device 10 a showingthe virtual object anchored to the anchor surface 16. In such cases, theprocessor within or coupled to second head mounted device 10 a mayreceive video data generated by the first head mounted device 10 a via awireless data link, and simply display the received video data as outputvideo images. In this application, the second user views the scene andthe anchored virtual object 14 a on the second head mounted display fromthe first user's perspective. Thus, in this implementation the seconduser may experience the first user's viewing experience like a movie orTV video. This first-user-perspective may be very useful in somesituations, such as when the first user is showing the second user howto do something for training or the second user is supervising theactivities of the first user. In this situation, the data sent over thedata link may be a simple video and audio stream, and the second headmounted device may simply project the received image onto the headmounted display with further processing or rendering of virtual objects.As described below, the video and image data from the first head mounteddevice may also be transmitted (e.g., via a network, server orpeer-to-peer wireless data link) to other computing devices, such assmartphones, tablet computers, laptop computers or large video displays,to enable others to view and share in the virtual/augmented realityexperience of the first user.

The second head mounted device 10 b may display the virtual object 14 bon the desk top surface 16. Alternatively, the second user may designateanother anchor surface on which the virtual object 14 should bedisplayed. When the virtual object 14 a is displayed on the first anchorsurface 16, the second head mounted device 10 b may modify the size andshape of the object so that it appears appropriate for the second user'sperspective. For example, if two users are viewing the same object oneither side of a table or desk, the object may appear upside down to oneof the users, just as if it were a real object. Alternatively, bothusers may view the same object in the same manner, so that both view theobject right side up.

For example, if the second user 10 b moves closer to the desktop anchorsurface 16, the virtual object 14 b may become larger. If the seconduser 10 b moves further from the desktop anchor surface 16, the virtualobject 14 b may become smaller. Further, if the second user 10 b rotatesthe display relative to the horizontal, the virtual object 14 b maysimilarly compensate for the rotation to appear anchored to the desktopanchor surface 16. As a further example, the user may move with his/herarms and hands in a manner that appears to interact with the object 14 bin order to close it or remove it from the display, such as sweeping anarm as if to brush the virtual object aside, closing fingers around thevirtual object, or punching the virtual object 14 b. As a furtherexample, a second user 10 b may execute gestures that attempt to pull orpush the virtual object 14 b and this pulling or pushing can beinterpreted as an input command. As mentioned above, recognizablegestures may be stored in a gesture dictionary which may store movementdata or patterns for recognizing gestures that may include pokes, pats,taps, pushes, guiding, flicks, turning, rotating, grabbing and pulling,two hands with palms open for panning images, drawing (e.g., fingerpainting), forming shapes with fingers (e.g., an “OK” sign), and swipes,all of which may be accomplished on or in close proximity to theapparent location of a virtual object in a generated display. In thismanner, users may interact with a virtual object 14 b anchored to asurface 16 using gestures that would be suitable for a real worldobject, thereby making interactions with virtual objects intuitive andeasy to learn.

In an embodiment described in more detail below, the head mounteddisplays may be replaced by projectors so that the images are projectedonto a surface. In such an embodiment, the operations and experience ofthe users may be similar to using head mounted displays as discussedabove. Also, as new image presentation technologies are developed, suchtechnologies may be implemented with the various embodiments to rendervirtual objects in conjunction with real world anchor surfaces andviewed by user from their perspectives.

FIG. 3 illustrates the virtual object 14 a presented to two userswearing head mounted devices 10 a, 10 b so the virtual object appears toeach user upside up (i.e., not upside down) even though the two usersare facing each other. This self-orientation of the virtual objectallows both users to view the object data (e.g., a bar chart) correctly.

As mentioned above, this presentation of the virtual object 14 a, 14 bmay be accomplished by the head mounted device 10 a capturing an imageof the table 16 utilizing a head mounted or body mounted camera. Theimage may be displayed in the head mounted display 10 a, or the tablemay be viewed through a semi-transparent display. The virtual object 14a for a first user may be displayed on the head mounted device 10 a. Thefirst user may formulate a signal using the first head mounted device 10a indicating that a second virtual object 14 b should be generated for asecond user. The first user may also provide an input indicating theanchor surface 16 in the captured image. The processor within or coupledto the head mounted device 10 a may calculate parameters includingdistance and orientation with respect to the head mounted or bodymounted camera 10 a that corresponds to the anchor desk top surface 16that was selected by the first user. The head mounted device 10 adisplays the generated virtual object 14 a so the virtual object 14 a isanchored to the anchor surface 16 and include the first orientation. Thefirst virtual object 14 a may then be displayed by the display on thefirst head mounted display 10 a as if connected to the first anchorsurface 16 a.

In FIG. 3, a second user wearing a second head mounted device 10 b sitsacross from the first user. The second head mounted device 10 b mayeither receive an input to select the desktop to be the anchor surface16 or may receive data from the first head mounted device 10 aidentifying the selected anchor surface 16. Using this information thesecond head mounted device 10 b may generate a display of the virtualobject 14 b reoriented to appear right side up and with the properperspective for the second user. To generate this display, the secondhead mounted device 10 b may receive data regarding the virtual objectto be rendered, such as its content and data regarding its general shapeand orientation. The second head mounted device 10 b may use the anchorsurface selected by the first user (or another anchor surface selectedby the second user) to determine a location, orientation and perspectivefor displaying the virtual object. This may include determining a propertop of the object, and an angle of projection of the object to match theanchor surface that results in the proper perspective of the renderedobject. Thus, as illustrated in FIG. 3, the second user views the samevirtual object 14 b anchored to the desk top surface 16 but right sideup from the second user's perspective.

In an alternative embodiment, first head mounted device 10 a or anothercomputing device in communication with both head mounted device maydetermine the proper orientation and perspective of the virtual object14 b for the second head mounted device 10 b. In this case, the firsthead mounted device 10 a may receive information sufficient to enablethe device to determine an orientation of the second user or of thesecond head mounted device 10 b. For example, images of the second usermay be captured by cameras on the first head mounted device andprocessed using anatomical models to determine the second user'sposition and viewing perspective. The first head mounted device 10 a maythen transmit the virtual object data to the second head mounted devicein a manner that correctly orients the virtual object 14 b for thesecond user 10 b based on the determined user orientation.

In various embodiments, the second user may provide inputs to the secondhead mounted device 10 b to control or change the orientation, size, andshape (e.g., perspective) of the virtual object 14 b. For example, thesecond user utilizing a detected gesture input command may drag thecorners of the virtual object 14 b to make it larger or smaller. Asanother example, the second user may minimize the virtual object 10 band open a new application or virtual object.

As illustrated in FIG. 4 two users viewing the same virtual object neednot be in the same room. For example, a first user wearing a first headmounted device 10 a at a first location, for example, Cambridge, UnitedKingdom, may view a virtual object 14 a (e.g., a presentation or graph)anchored on a first anchor surface 16 a (e.g., a wall) and collaborateregarding the virtual object with a second user wearing a second headmounted device 10 b at a second location, for example, San Diego,Calif., who may view the virtual object 14 b anchored to a verydifferent anchor surface 16 b, such as a table top. The head mounteddevice 10 a may communicate with the second head mounted device 10 b viaany computer network known in the art that interconnects the devices 10a and 10 b and that allows users to share resources by wired andwireless communication links.

For example, the first head mounted device 10 a may transmit via acommunication network a signal correctly orienting the second virtualobject 14 b for the second head mounted device 10 b and a signalgenerating the oriented second virtual object 14 b for the second user10 b. The second head mounted device 10 b may receive the signal fromthe communication network and may generate the second virtual object 14b for the second user 10 b.

In another embodiment, images of the first and/or second users can becaptured over time by the head mounted or body mounted camera and usedto determine an appropriate orientation for virtual objects. Ananatomical analysis can be applied to the captured images to determinethe current posture and orientation of the user, such as with respect toa designated anchor surface. For example, the images may be compared toanatomical data stored in memory to detect a body part within the imagesand determine an orientation of the user. The second virtual object 14 bfor the second head mounted device 10 b may be oriented correctly basedon the detected body part in the images of either the first user (i.e.,to present the image to the second user in the manner it appears to thefirst user) or the second user (i.e., to present the image in a mannersuitable for the second user's posture). For example, images of thesecond user may be captured, tracked and compared to anatomical datastored in memory where the second user's head, shoulders, arms, torso,legs or any other body part or a portion thereof may be detected. Thesecond head mounted device 10 b may orient the virtual object based onthe detected body parts. The second virtual object 14 b may be orientedin the first orientation and displayed.

A time period later, the second user may move and the first orientationmay not be appropriate based on the different orientation and the user'snew location. For example, parameters may be calculated includingdistance and orientation with respect to the second head mounted or bodymounted camera 10 b that correspond to the second anchor surface 16 bwith the recognized object located in the image. An image size and anorientation may be determined that correspond to the determinedorientation for the second virtual object 14 b after such a movement.The second head mounted device 10 b may then display the generatedsecond virtual object 14 b so the second virtual object 14 b appearsanchored and connected to the second anchor surface 16 b. For example,the detected body part may be utilized to determine where the seconduser is looking and the second virtual object 14 b may be orientedcorrectly based on where the second user is looking.

In another embodiment, the detected body parts may be utilized todetermine whether the second user is sitting or standing. Based on thedetermination, the second virtual object 14 b may be generated to beanchored to and oriented on a planar surface adjacent to the seconduser. If it is determined based on the detected body part that thesecond user is standing, the second virtual object 14 b may be generatedon a vertical surface adjacent to the second user.

In the illustration of FIG. 4, the anchor surface 16 a is a wall locatedwithin the image displayed by the head mounted device 10 a. For thefirst user, the head mounted device 10 a may generate a virtual object14 a and display the virtual object 14 a as though fixed or connected tothe wall surface 16 and oriented so that it is right side up for thefirst user. The head mounted device 10 a processor may calculateparameters for rendering the virtual object including distance andorientation from the head mounted device 10 a to the wall 16 selected bythe first user as the anchor surface. The head mounted device 10 a mayinclude or be coupled to a mobile CPU that is coupled to a transceiverconfigured to communicate with a router to communicate data via anetwork, such as the Internet, to the second head mounted device 10 b.Similarly, the second head mounted device 10 b may also include atransceiver to communicate with a network to send/receive data via anetwork, such as the Internet, an intranet, a wireless network, or anyother appropriate telecommunications network.

Either of the first and second head mounted devices 10 a, 10 b mayinitiate a collaboration session and send a request for such a sessionto the other via a network, such as the Internet, such as by sendingmessages to the IP address associated with the other head mounteddevice. In another embodiment, the head mounted device 10 a maycommunicate with the second head mounted device 10 b via a server thatincludes program instructions to execute a service to share resourcesamong the two devices 10 a and 10 b.

As an example, a collaboration session between a first and second userboth equipped with head mounted devices may proceed as follows. When anasset or virtual object is added or anchored to a surface by one user,that user's head mounted device may transmit a message to the otheruser's head mounted device indicating a preferred placement of the newasset or object in a manner that is relative to an existing asset orvirtual object already displayed on a previously selected anchorsurface. This preferred placement of the new asset or object may then beoverridden by the system automatically or users through user inputs(e.g., recognized movement gestures) if there is inadequate space forboth objects on the original anchor surface. Also or alternatively, oneor both of the users may execute an input or gesture in order tomanually reposition the new asset or virtual object to position orsurface of their preference.

In another embodiment, the first head mounted device 10 a may deliver amessage to the second head mounted device 10 b data indicating apreferred or suggested placement of the virtual object 14 b. Forexample, the first user may indicate that the preferred placement of thevirtual object 14 b should occur on a desk 16 b or on a wall. Forexample, a message may be sent from the first head mounted device 10 aindicating “Wall Surface Preferred” or “Desktop Surface Preferred” tothe second head mounted device 10 b. The second head mounted device 10 bmay generate a second virtual object 14 b on the display. The messagemay also include one or more program instructions to correctly orientthe virtual object 14 for the second user. The second head mounteddevice 10 b may receive an input to select the desktop as the anchorsurface 16 b or may receive data from the first head mounted device 10 athat the anchor surface 16 is selected as a wall. In the case of thelatter alternative, a processor within or coupled to the second headmounted device 10 b may detect a comparable wall within images obtainedby a camera on the device. Program instructions may also be communicatedto the processor for the user to accept or reject the suggested anchorsurface from another user. The head mounted display 10 b may receive aninput command from the second user in the form of a spoken audio signalto accept or to reject the suggested anchor surface. The second headmounted device 10 b may receive the message indicating a preferredplacement of the second virtual object 14 b and the second user mayprovide an input to the second head mounted device 10 b confirming orrejecting the preferred placement of the second virtual object 14 b. Forexample, the second user may provide an input indicating a desire torotate and reorient the second virtual object 14 b. The processor maydetermine distance and orientation based on the input for the secondvirtual object 14 and may generate the second virtual object 14 b withthe desired orientation.

The second head mounted device 10 b may display the virtual object 14 bon the desk top surface 16 b at the second location. The second user mayfurther provide inputs to change the orientation, size, and shape of thevirtual object 10 b to suit the anchor surface 16 b. The virtual object14 b appears the same virtual object 14 a and collaboration may occurbetween the users. Changes made at one location by one user may updateand be shown at both the first and the second locations, i.e., changesto virtual object 14 a in one location will be reflected in the virtualobject 14 b displayed in the other location. For example, the seconduser may delete a bar chart on the virtual object 14 b and the same barchart on the virtual object 10 a may also be deleted at the firstlocation.

In yet another embodiment, a third user wearing a third head mounteddevice may join in the collaboration and anchor a displayed virtualobject on a surface selected by the third user or suggested by either ofthe first or second users. Also, as described above, image datagenerated by one or more of the head mounted devices involved in acollaboration may be transmitted to and displayed on other head mounteddevices or other computing devices, such as smartphones or tablets usedby non-participants. This embodiment enables others to view thecollaboration in a virtual or augmented reality experience. Othersviewing a collaboration session wearing head mounted devices or usinganother mobile device such as a smartphone or tablet computer may notonly see the virtual objects and user interactions with them, but havelimited interaction capabilities with the virtual augmentations seen byone of the head mounted device users. This limited interaction mayinclude touching the augmentation to cause an effect, defining aninteractive area or anchor point on the physical surface (effectivelyadding a new augmentation to the shared experience), and interactingwith the shared mixed reality scene via gestural and/or audio inputs.Any of such changes in the virtual experience may be reflected in thehead mounted displays worn by the other users.

In an embodiment, the second user wearing the second head mounted device10 b may provide an input to summon a new virtual object from a personaldata space (e.g., cloud or mobile device) and add the new virtual objectto a shared display so the first user also sees it in the first headmounted device 10 a. In an embodiment, the first head mounted device 10a may receive a prompt which informs the user that a third virtualobject is present and requests a user input or command to accept anddisplay the third virtual object. The user may select a new physicalsurface to anchor the new virtual object to, or may accept the anchorsurface selected by the second user.

In an embodiment, the processor within or coupled to the head mounteddevice 10 a may receive an input indicating a standby location for avirtual object 14. The standby location may be a location where aninitial condition of the virtual object 14 can be displayed as notanchored or fixed to a real world surface. The standby location may be alocation where the virtual object 14 is “free floating” on the displayunconnected to a real world object. Alternatively, the standby locationcan be a different anchor surface 16 located in the image, for example,on a ceiling, or a repeated and familiar location for the user. Thestandby location may be a place or surface where the object can be“stored” until the user takes further action to anchor the virtualobject 14 to an anchor surface. The processor within or coupled to thehead mounted device 10 a may calculate parameters including distance andorientation with respect to the head mounted or body mounted camera thatcorresponds to the standby location and the processor may display thevirtual object 14 with the proper perspective for the standby location.The standby location may be displayed as free floating and may appear tobe movable and unconnected to surfaces in the image as the user turnshis/her head and moves about the room.

In an embodiment, the processor within or coupled to the head mounteddevice 10 a may capture an image with a head mounted or body mountedcamera or cameras which may be full-color video cameras. Distances toobjects within the imaged scene may be determined via trigonometricprocessing of two (or more) images obtained via a stereo cameraassembly. Alternatively or in addition, the head mounted device mayobtain spatial data (i.e., distances to objects in the images) using adistance sensor which measures distances from the device to objects andsurfaces in the image. In an embodiment, the distance sensor may be aninfrared light emitter (e.g., laser or diode) and an infrared sensor. Inthis embodiment, the head mounted device may project infrared light aspulses or structured light patterns which reflect from objects withinthe field of view of the device's camera. The reflected infrared laserlight may be received in a sensor, and spatial data may be calculatedbased on a measured time or phase shift between the emitted and receivedlight. In another embodiment, the distance sensor may be an ultrasoundemitter and receiver. In this embodiment, the head mounted device mayemit ultrasound pulses, which reflect from objects within the field ofview of the device's camera. An ultrasound receiver senses reflectedultrasound, and calculates spatial data based on a measured time betweensound emissions and received reflected ultrasound. In a thirdembodiment, distances to objects may be estimated by measuring changesin angles in subsequent images in a video stream as the user's headmoves and applying trigonometric analyses to the sequence of images. Byincluding motion and/or relative position sensors in the head mounteddevice, changes in viewing angle may be correlated to changes in cameraposition, from which object/surface distances and locations may becalculated using trigonometric algorithms. Any combination of thesedistance-measuring methods may be combined to further refine thedistance measurements obtained by the head mounted device.

In an embodiment, the processor within or coupled to the head mounteddevice 10 may store the spatial data in memory and may build athree-dimensional map of objects and surfaces within the vicinity of thedevice based on the captured video and the measured or calculatedspatial data. This three-dimensional map may be shared with other headmounted devices within the vicinity or connected via a network, and/oruploaded to a server. Sharing such three-dimensional map data may assistother head mounted devices 10 b (or more specifically the processorswithin or coupled to the other head mounted devices) to render virtualobjects anchored to a designated anchor surface without having to imageand process the room. Sharing of such three-dimensional map data mayalso enable multiple head mounted devices to collaboratively create anoverall map of the vicinity of the multiple users, and reduce theworkload imposed on any one head mounted device by avoiding the need toscan the same environment multiple times (i.e., the three-dimensionalmap only needs to be generated once). In a further application, thesharing of image and spatial data, such as a three-dimensional map ofobjects in the vicinity of a first user, may enable other users to viewthe same images as the first user even when they are located far away.For example, a second device may utilize the map where data can betransmitted that corresponds to captured spatial data so another userwearing a second head mounted device may display the virtual object 14in the same manner as the first user.

As explained above, the display associated with the head mounted device10 a or 10 b may be partially transparent or may substantially occludeall or a portion of the user's view. In another embodiment, the headmounted device 10 a display may be positioned over only one eye so thatit occludes only a portion of the user's vision, such as in the form ofa monocular display so one user's eye is able to view the roomunobstructed. In another embodiment, the head mounted device 10 a or 10b may output the virtual object 14 a or 14 b content to a computingdevice display so that the images obtained by the head or body mountedcameras and any virtual objects may be viewed together on a conventionalcomputer display.

As mentioned above, the head mounted displays may be replaced byprojectors that project images onto a surface, or by other imagegeneration technologies that may be developed. In such an embodiment,the operations and experience of the users may be similar to that usinghead mounted displays as discussed above.

FIG. 5A illustrates components that may be included in embodiments ofhead mounted devices 10. FIG. 5B illustrates how head mounted devices 10may operate as part of a system in which a sensor array 500 may providedata to a mobile processor 507 that performs operations of the variousembodiments described herein, and communicates data to and receives datafrom a server 564. It should be noted that the processor 507 headmounted device 10 may include more than one processor (or a multi-coreprocessor) in which a core processor may perform overall controlfunctions while a coprocessor executes applications, sometimes referredto as an application processor. The core processor and applicationsprocessor may be configured in the same microchip package, such as amulti-core processor, or in separate chips. Also, the processor 507 maybe packaged within the same microchip package with processors associatedwith other functions, such as wireless communications (i.e., a modemprocessor), navigation (e.g., a processor within a GPS receiver), andgraphics processing (e.g., a graphics processing unit or “GPU”).

The head mounted device 10 may communicate with a communication systemor network that may include other computing devices, such as personalcomputers and mobile devices with access to the Internet. Such personalcomputers and mobile devices may include an antenna 551, atransmitter/receiver or transceiver 552 and an analog to digitalconverter 553 coupled to a processor 507 to enable the processor to sendand receive data via a wireless communication network. For example,mobile devices, such as cellular telephones, may access the Internet viaa wireless communication network (e.g., a Wi-Fi or cellular telephonedata communication network). Such wireless communication networks mayinclude a plurality of base stations coupled to a gateway or Internetaccess server coupled to the Internet. Personal computers may be coupledto the Internet in any conventional manner, such as by wired connectionsvia an Internet gateway (not shown) or by a wireless communicationnetwork.

Referring to FIG. 5A, the head mounted device 10 may include a scenesensor 500 and an audio sensor 505 coupled to a control system processor507 which may configured with a number of software modules 510-550. Inan embodiment, the processor 507 or scene sensor 500 may apply ananatomical feature recognition algorithm to the images to detect one ormore anatomical features. The processor 507 associated with the controlsystem may review the detected anatomical features in order to recognizeone or more gestures and process the recognized gestures as an inputcommand. For example, as discussed in more detail below, a user mayexecute a movement gesture corresponding to an input command, such aspointing a finger at the virtual object to close the virtual object. Inresponse to recognizing this example gesture, the processor 507 mayremove the virtual object from the display. As another example, the usermay touch the forefinger to the thumb on one hand to form the “OK” signin order to confirm an instruction or option presented on the display.

The scene sensor 500, which may include stereo cameras, orientationsensors (e.g., accelerometers and an electronic compass) and distancesensors, may provide scene-related data (e.g., images) to a scenemanager 510 implemented within the processor 507 which may be configuredto interpret three-dimensional scene information. In variousembodiments, the scene sensor 500 may include stereo cameras (asdescribed below) and distance sensors, which may include infrared lightemitters for illuminating the scene for an infrared camera. For example,in an embodiment illustrated in FIG. 5A, the scene sensor 500 mayinclude a stereo red-green-blue (RGB) camera 503 a for gathering stereoimages, and an infrared camera 503 b configured to image the scene ininfrared light which may be provided by a structured infrared lightemitter 503 c. As discussed below, the structured infrared light emittermay be configured to emit pulses of infrared light that may be imaged bythe infrared camera 503 b, with the time of received pixels beingrecorded and used to determine distances to image elements usingtime-of-flight calculations. Collectively, the stereo RGB camera 503 a,the infrared camera 503 b and the infrared emitter 503 c may be referredto as an RGB-D (D for distance) camera 503.

The scene manager module 510 may scan the distance measurements andimages provided by the scene sensor 500 in order to produce athree-dimensional reconstruction of the objects within the image,including distance from the stereo cameras and surface orientationinformation. In an embodiment, the scene sensor 500, and moreparticularly an RGB-D camera 503, may point in a direction aligned withthe field of view of the user and the head mounted device 10. The scenesensor 500 may provide a full body three-dimensional motion capture andgesture recognition. The scene sensor 500 may have an infrared lightemitter 503 c combined with an infrared camera 503 c, such as amonochrome CMOS sensor. The scene sensor 500 may further include stereocameras 503 a that capture three-dimensional video data. The scenesensor 500 may work in ambient light, sunlight or total darkness and mayinclude an RGB-D camera as described herein. The scene sensor 500 mayinclude a near-infrared (NIR) pulse illumination component, as well asan image sensor with a fast gating mechanism. Pulse signals may becollected for each pixel and correspond to locations from which thepulse was reflected and can be used to calculate the distance to acorresponding point on the captured subject.

In another embodiment, the scene sensor 500 may use other distancemeasuring technologies (i.e., different types of distance sensors) tocapture the distance of the objects within the image, for example,ultrasound echo-location, radar, triangulation of stereoscopic images,etc. As discussed above, in an embodiment, the scene sensor 500 mayinclude a ranging camera, a flash LIDAR camera, a time-of-flight (ToF)camera, and/or a RGB-D camera 503, which may determine distances toobjects using at least one of range-gated ToF sensing, RF-modulated ToFsensing, pulsed-light ToF sensing, and projected-light stereo sensing.In another embodiment, the scene sensor 500 may use a stereo camera 503a to capture stereo images of a scene, and determine distance based on abrightness of the captured pixels contained within the image. Asmentioned above, for consistency any one or all of these types ofdistance measuring sensors and techniques are referred to hereingenerally as “distance sensors.” Multiple scene sensors of differingcapabilities and resolution may be present to aid in the mapping of thephysical environment, and accurate tracking of the user's positionwithin the environment.

The head mounted device 10 may also include an audio sensor 505 such asa microphone or microphone array. An audio sensor 505 enables the headmounted device 10 to record audio, and conduct acoustic sourcelocalization and ambient noise suppression. The audio sensor 505 maycapture audio and convert the audio signals to audio digital data. Aprocessor associated with the control system may review the audiodigital data and apply a speech recognition algorithm to convert thedata to searchable text data. The processor may also review thegenerated text data for certain recognized commands or keywords and userecognized commands or keywords as input commands to execute one or moretasks. For example, a user may speak a command such as “anchor virtualobject” to anchor the displayed virtual object on a selected surface.For example, the user may speak “close application” to close anapplication displayed on the display.

The head mounted device 10 may also include a display 540. The display540 may display images obtained by the camera within the scene sensor500 or generated by a processor within or coupled to the head mounteddevice 10. In an embodiment, the display 540 may be a micro display. Thedisplay 540 may be a fully occluded display. In another embodiment, thedisplay 540 may be a semitransparent display that can display images ona screen that the user can see through to view the surrounding room. Thedisplay 540 may be configured in a monocular or stereo (i.e., binocular)configuration. Alternatively, the head-mounted device 10 may be a helmetmounted display device, worn on the head, or as part of a helmet, whichmay have a small display 540 optic in front of one eye (monocular) or infront of both eyes (i.e., a binocular or stereo display). Alternatively,the head mounted device 10 may also include two display units 540 thatare miniaturized and may be any one or more of cathode ray tube (CRT)displays, liquid crystal displays (LCDs), liquid crystal on silicon(LCos) displays, organic light emitting diode (OLED) displays, Mirasoldisplays based on Interferometric Modulator (IMOD) elements which aresimple micro-electro-mechanical system (MEMS) devices, light guidedisplays and wave guide displays, and other display technologies thatexist and that may be developed. In another embodiment, the display 540may comprise multiple micro-displays 540 to increase total overallresolution and increase a field of view.

The head mounted device 10 may also include an audio output device 550,which may be a headphone and/or speaker collectively shown as referencenumeral 550 to output audio. The head mounted device 10 may also includeone or more processors that can provide control functions to the headmounted device 10 as well as generate images, such as of virtual objects14. For example, the device 10 may include a core processor, anapplications processor, a graphics processor and a navigation processor.Alternatively, the head mounted display 10 may be coupled to a separateprocessor, such as the processor in a smartphone or other mobilecomputing device. Video/audio output may be processed by the processoror by a mobile CPU, which is connected (via a wire or a wirelessnetwork) to the head mounted device 10. The head mounted device 10 mayalso include a scene manager block 510, a user control block 515, asurface manager block 520, an audio manager block 525 and an informationaccess block 530, which may be separate circuit modules or implementedwithin the processor as software modules. The head mounted device 10 mayfurther include a local memory and a wireless or wired interface forcommunicating with other devices or a local wireless or wired network inorder to receive digital data from a remote memory 555. Using a remotememory 555 in the system may enable the head mounted device 10 to bemade more lightweight by reducing memory chips and circuit boards in thedevice.

The scene manager block 510 of the controller may receive data from thescene sensor 500 and construct the virtual representation of thephysical environment. For example, a laser may be used to emit laserlight that is reflected from objects in a room and captured in a camera,with the round trip time of the light used to calculate distances tovarious objects and surfaces in the room. Such distance measurements maybe used to determine the location, size and shape of objects in the roomand to generate a map of the scene. Once a map is formulated, the scenemanager block 510 may link the map to other generated maps to form alarger map of a predetermined area. In an embodiment, the scene anddistance data may be transmitted to a server or other computing devicewhich may generate an amalgamated or integrated map based on the image,distance and map data received from a number of head mounted devices(and over time as the user moved about within the scene). Such anintegrated map data made available via wireless data links to the headmounted device processors. The other maps may be maps scanned by theinstant device or by other head mounted devices, or may be received froma cloud service. The scene manager 510 may identify surfaces and trackthe current position of the user based on data from the scene sensors500. The user control block 515 may gather user control inputs to thesystem, for example audio commands, gestures, and input devices (e.g.,keyboard, mouse). In an embodiment, the user control block 515 mayinclude or be configured to access a gesture dictionary to interpretuser body part movements identified by the scene manager 510. Asdiscussed above a gesture dictionary may store movement data or patternsfor recognizing gestures that may include pokes, pats, taps, pushes,guiding, flicks, turning, rotating, grabbing and pulling, two hands withpalms open for panning images, drawing (e.g., finger painting), formingshapes with fingers (e.g., an “OK” sign), and swipes, all of which maybe accomplished on or in close proximity to the apparent location of avirtual object in a generated display. The user control block 515 mayalso recognize compound commands. This may include two or more commands.For example, a gesture and a sound (e.g. clapping) or a voice controlcommand (e.g. ‘OK’ detected hand gesture made and combined with a voicecommand or a spoken word to confirm an operation). When a user control515 is identified the controller may provide a request to anothersubcomponent of the device 10.

The head mounted device 10 may also include a surface manager block 520.The surface manager block 520 may continuously track the positions ofsurfaces within the scene based on captured images (as managed by thescene manager block 510) and measurements from distance sensors. Thesurface manager block 520 may also continuously update positions of thevirtual objects that are anchored on surfaces within the captured image.The surface manager block 520 may be responsible for active surfaces andwindows. The audio manager block 525 may provide control instructionsfor audio input and audio output. The audio manager block 525 mayconstruct an audio stream delivered to the headphones and speakers 550.

The information access block 530 may provide control instructions tomediate access to the digital information. Data may be stored on a localmemory storage medium on the head mounted device 10. Data may also bestored on a remote data storage medium 555 on accessible digitaldevices, or data may be stored on a distributed cloud storage memory,which is accessible by the head mounted device 10. The informationaccess block 530 communicates with a data store 555, which may be amemory, a disk, a remote memory, a cloud computing resource, or anintegrated memory 555.

FIG. 6 illustrates an embodiment method 600 for rendering a virtualobject on an anchored surface based on a user input. Method 600 may beimplemented in a computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 600. In method 600, the processor may commence upon receivingdata from the head mounted device sensors regarding an orientation ofthe head mounted device in block 601. In block 602, the processor mayreceive data from stereo cameras on the head mounted device or on theshoulders or body of the user. In block 603, the processor may calculatedistance and orientation data of objects within the received stereoimage. In block 604, the processor may process the content for displayusing data from the distance and orientation sensors, and the camera.

In determination block 605, the processor may review user inputs orgesture commands to determine whether to anchor a virtual object in theimage. For example, a user input may be received from an input device,such as a keypad or touchscreen input. As another example, a user inputmay be received in the form of a detected gesture recognized in asequence of images, such as a gesture indicating a desire to anchor thevirtual object on an indicated surface located within the image. A thirdexample of a user input may be an audio command that may be received bya microphone on the head mounted device. A user input may be a commandindicating that an input for anchoring the virtual object to aparticular location should be received from a second user's head mounteddevice. For example, the input may be received from a second headmounted device 10 b during a collaboration session.

If the processor determines that it has received an input to anchor thevirtual object at a particular location (e.g., an anchor surface 16)(i.e., determination block 605=“Yes”), the processor may determine thedistance and orientation of the virtual object relative to the anchorsurface from the user's position in block 606. For example, the headmounted device 10 may determine that the anchor surface 16 includes acontour and a certain distance from the user. In general, virtualobjects 14 may be three-dimensional virtual objects, such as in the formof three-dimensional mathematical models (e.g., spheres, cones,cylinders, etc.), three-dimensional data sets (e.g., data tablesdefining vertices, dimensions, and other coordinate and material data),or combinations of both, which a processor can use to generate an imageof the object. In order for the processor to be able to display thevirtual object 14 connected on the anchor surface 16, the virtual object14 may be adjusted to a size, viewing perspective and orientation tosuperimpose the virtual object on the anchor surface 16. For example, athree-dimensional virtual object will need to be rendered as atwo-dimensional image to be presented on a conventional display, or astwo slightly different two-dimensional images to be presented on 3Ddisplays on a head mounted device.

In block 607, the processor may process the virtual object to place thevirtual object in a virtual space and with a perspective, orientationand size that causes it to appear connected to the anchored surface. Inblock 608, the processor may render the virtual object on the headmounted device display.

For example, the display may be a semitransparent display where certainfeatures of the image are viewable through the display. The virtualobject 14 may be superimposed or added to the features of the image onthe display so the virtual object 14 remains anchored to, or fixed to,the desired anchored surface as the user moves. Alternatively, thevirtual object 14 a may be anchored in free space. For example, thevirtual object 14 a may be located outside of a window, or the virtualobject 14 a may be suspended in midair or a distance above the ground.

If the processor determines that it has not received an input to anchorthe virtual object (i.e., determination block 605=“No”), which mayindicate that the user is not ready or does not want to anchor thevirtual object to an anchor surface, the processor may render thevirtual object in a standby location. Alternatively, the user may choosenot to render the virtual object 14 in block 609.

The processes of rendering the virtual object within the scene, such ason the anchor surface, may be performed continuously so that the virtualobject appears stationary even as the user turns his/her head and movesabout. As mentioned above, the process of orienting the head mounteddevice with respect to the surroundings, and determining distances toand relative orientation of various surfaces may be accomplishedcontinuously by monitoring orientation sensors (e.g., accelerometers andgyroscopes) and triangulating its own position and angle of view as thedevice moves in space by constantly referencing shifts in the topographyof the scanned three-dimensional (3D) environment, thus performingSimultaneous Location and Mapping (SLAM) operations.

FIG. 7A illustrates an embodiment method 700 for rendering an image withan anchored virtual object. Method 700 may be implemented in a computingdevice having a processor with processor-executable instructions toperform the operations of the method 700. In method 700, the virtualobject 14 may be anchored as the position of the user moves through anarea, such as a room. For example, the user may walk and move about inthe room, but the displayed location of the virtual object on a surfacein the room may remain fixed. In method 700 in block 701, the processormay commence operation by receiving user position data such as thecoordinates of a user's location, information related to the user'sposition within a room (e.g., distances from various surfaces), andinformation related to an orientation of the user's head. The positionaldata received in block 701 may be calculated from a range of sensordata, including distance sensors (e.g., an infrared emitter/receiverconfigure to calculate the round trip time of emitted infrared signalsbouncing off of surfaces within the viewing perspective of the user witha fast gating mechanism). Alternatively or additionally, positional datamay be provided by a GPS receiver unit (which may return geographiccoordinates), triangulation of RF signals from transmitters at knownlocations (e.g., cellular telephone towers), inputs received from asecond head mounted device (e.g., providing separation distance and/orinformation regarding relative positions of the two users), etc.Position data may also be calculated based on scene image and distancesensor data, or downloaded from a server in a cloud computingarrangement. Alternatively or in addition, the user may inputcoordinates of the user's location, such as by typing in a physicaladdress and/or a room number, or touching or clicking on a map displayto indicate the user's position on the map. In block 702, the processormay calculate distance and orientation data of objects within thereceived camera image. In block 703, the processor may calculate theposition of the user relative to the anchored virtual object.

The scene sensor 500 of FIG. 5A may feature a near-infrared (NIR) pulseillumination component. The scene sensor 500 may also have an imagesensor with a fast gating mechanism. Other image sensors may also beimplemented within the scene sensor 500 as described herein. Based onthe known speed of the infrared pulses, the coordinates, and the timingof infrared pulse wave emissions from the illuminator with the gating ofthe sensor 500, a signal reflected from within a desired distance rangeis captured. In this embodiment, the time of arrival of infrared pulsesmay be collected for each pixel and used to calculate the distance fromthe sensor to each point in the image. Such processing of the image maybe performed using a remote processor that is connected to the scenesensor 500 via a computer network. The method 700 is not limited tobeing implemented on a head mounted display and may be used with anothercomputing device, such as a mobile communication device (e.g., asmartphone) or a fixed computing device (e.g., a personal computer or aserver) in communication with the head mounted display or body mountedsensors.

In determination block 704, the processor may determine whether theposition of the user or the anchor surface has changed. For example, theuser may anchor the image on a surface such as a wall or in free spaceand the user may walk away from the wall during collaboration withanother user, thus changing position. If the position has changed (i.e.,determination block 704=“Yes”) which indicates the user has moved awayfrom the anchored virtual object or the anchor surface has moved, theprocessor may determine and calculate a change of the anchored virtualobject based on the new position in block 705. In block 706, theprocessor may render the image with the anchored virtual object on thehead mounted display. If the processor determines that the virtualobject has not moved (i.e., determination block 704=“No”), the anchoredvirtual object may be rendered unchanged. The processor may render thevirtual object in the image for the current position on the head mounteddisplay in block 706.

FIG. 7B illustrates an embodiment method 720 for moving a virtual objectbetween anchor surfaces. Method 720 may be implemented in a computingdevice having a processor configured with processor-executableinstructions to perform the operations of the method 720. A first inputmay be provided anchoring a virtual object on a first anchor surface.Later, the processor may receive a second input to anchor the virtualobject on a second different anchor surface. In method 720, theprocessor may commence operation by capturing an image in block 707. Inblock 708, the processor may capture spatial data and recognize objectswithin the image. In block 709, the processor may determine a distanceparameter of the objects in the image. In block 710, the processor mayreceive an input indicating an anchor surface within the image.

For example, the processor may receive an input from an input device, ormay detect a gestural input, or may receive a voice command to anchorthe virtual object 14 within the image. In block 711, the processor maygenerate the virtual object 14 that is displayed, such as by using anobject model and/or an object image data set. In block 712, theprocessor may calculate parameters including distance and orientationcorresponding to the anchor surface, and may adjust the image of thevirtual object consistent with those parameters to coincide with theviewer's perspective. In block 713, the processor may display thevirtual object on the head mounted display so the virtual object appearsto be connected to the anchor surface.

The user may choose to change the anchor surface due to a preference orneeds of the collaborative process with other users. The variousembodiments enable each user to select the same or different anchorsurfaces on which to view virtual objects 14. So, in determination block714 the processor may monitor user inputs (e.g., buttons, recognizedmovement gestures and/or spoken commands) and determine whether a secondinput is received indicating a second different anchor surface for thevirtual object 14. If a second input is not received (i.e.,determination block 714=“No”), the processor may return to block 711 tocontinue generating the virtual object in the image connected to thefirst anchor surface.

If a second input is received (i.e., determination block 714=“Yes”), inblock 715 the processor may detect one or more inputs that indicate anew anchor surface. When a new anchor surface is indicated, theprocessor may calculate parameters including distance and orientationcorresponding to the new anchor surface that was selected in block 715,and in block 716, the processor may generate images presented on thehead mounted display so that the virtual object 14 appears to be movedfrom the first anchor surface to the second new anchor surface. In block717, the processor may modify the image of virtual object to correspondto changes of position, and thus viewing perspective, of the user. Forexample, if the user walks closer the virtual object image size mayincrease. If the user walks a distance away from the virtual object, thevirtual object's image size may decrease. The processes of modifyingimages of the virtual object to match changes in the user's viewingperspective in block 717, as well as monitoring for user inputs indetermination block 714 may continue to be performed in a continuousprocess throughout a collaboration session.

FIG. 8A illustrates an embodiment method 800 for rendering an image withan anchored virtual object for two users in an area, such as, forexample, a room. In this embodiment, the displayed anchored virtualobject 14 remains anchored to the selected anchor surface. Method 800may be implemented in a computing device having a processor configuredwith processor-executable instructions to perform the operations of themethod 800.

In method 800, the processor may commence operation by processing animage using data from distance sensors, camera data and calculateddistance data in block 801. In block 802, the processor may calculateparameters and may fix the virtual object to the anchor surface and fixthe virtual object in the image that is presented on the head mounteddisplay. In block 803, the processor may render the image for displaywith the virtual object. In block 804, the processor may control atransmitter/receiver or transceiver to transmit image and/or virtualobject data to the second head mounted display. In block 804, the firsthead mounted device may transmit the three-dimensional model and/orthree-dimensional data set defining the virtual object to the secondhead mounted device in a format that enables the second device togenerate an image of the same virtual object. Also in block 804, theprocessor may transmit a map including positional coordinate informationto the second head mounted device. The head mounted device may uploadand/or download the room coordinate map from a server, or may transmit amap directly to a second head mounted device. In block 805, a secondprocessor may render an image of the virtual object for projection onthe second head mounted display. The image may include the virtualobject anchored or fixed to the anchor surface, which was selected bythe first user in block 802. For example, the first user may anchor thevirtual object on a desktop surface. In block 805, the processor mayrender the image. In the rendered image, the virtual object will appearon the desktop surface since it was selected by the first user in block802 by the input.

In the various embodiments, a second head mounted device may display animage of the virtual object anchored to either the same anchor surfaceas designated by the user of the first head mounted device, or to adifferent anchor surface identified by the second user. In block 806,for the second head mounted device, the second processor may determine alocation and orientation where the second user is looking (which isrelated to the orientation of the head mounted device) in order todetermine where within the displayed image an anchored virtual objectshould be rendered. For example, if the second user is looking at theanchor surface designated by the first user, then the second headmounted device may render an image of the virtual object in the headmounted display. However, if the second user is looking away from theanchor surface, the second head mounted device would not render an imageof the virtual object since the anchor surface would be outside theuser's field of view presented on the head mounted display.

Determining the orientation of the user's head and the head mounteddevice may be accomplished by capturing images by the cameras worn bythe second user and data from orientation sensor (e.g., an inclinometer,gyroscopes and/or accelerometers) of the second user's head mounteddevice, and processing this information to infer the viewing angle andperspective of the second user. As discussed above, this process may beaccomplished continuously such that each head mounted device cantriangulate its own position and angle of view as it moves in space byconstantly referencing the shifts in the topography of scannedthree-dimensional (3D) environment, thus performing SimultaneousLocation and Mapping (SLAM) operations. Also, if a portion of the seconduser's anatomy is captured in an image from the second user's headmounted or body mounted cameras, this may also include applying adetection algorithm to the captured image to recognize a body part, andfrom that infer where the body part's field of view is located.

In block 807, the processor may process the virtual object data togenerate image data for use in rendering an image of the object. In thisoperation, the processor may render the virtual object appropriatelyfrom the second user's perspective and so the second user cancomfortably read content and view the virtual object. In block 808, theprocessor may generate an image in the second head mounted display basedon the location and the orientation of the anchor surface with respectto the user.

In determination block 809, the processor may determine whether thesecond user would like to determine a new anchor surface. Thedetermination may be from receiving an input signal indicating a desireto anchor the virtual object 14 in the image. For example, the seconduser may wish to anchor the image on a different surface than a currentanchor surface. For example, the image may be fixed to a wall during acollaborative process with multiple users and then moved to a desktop.

As mentioned above, the user of the second head mounted device may electto change the anchor surface, and thus move the image of the virtualobject to another surface designated by the second user. If the headmounted device processor receives a user input indicating a desire toanchor the virtual object on a new surface (i.e., determination block809=“Yes”) which indicates the second user desires to determine a newanchor for the virtual object 14, the processor may determine distanceand orientation parameters of a new anchor of the virtual object on asecond predetermined surface in block 810. In block 811, the processormay change or modify the virtual object to the parameters of the newanchor surface. The processor may process the virtual object 14 based ona distance and an orientation in block 812. In block 813, the secondprocessor may render the image on the head mounted display with theanchored virtual object fixed on the second anchor surface.

If the processor does not receive a user input indicating a desire toanchor the virtual object on a new surface (i.e., determination block809=“No”), the processor may process the virtual object for rendering inblock 807 with no change in anchor position but rendered for the seconduser's perspective. In a similar manner, the first or the second usermay add another virtual object and anchor the additional virtual objecton a third anchor surface.

FIG. 8B illustrates an embodiment method 815 for correctly orienting ananchored virtual object in an image that is output on a display of afirst user from the first user's point of view, and on another displayfor a second user's point of view. Method 815 may be implemented in acomputing device having a processor configured with processor-executableinstructions to perform the operations of the method. In method 815 inblock 820, the processor may receive image data as the user scans a roomand process the image data to identify/recognize surfaces, includingsurfaces suitable for serving as an anchor for virtual objects. Forexample, the processor may capture images from a head mounted or bodymounted device. The processor of such a head or body mounted device mayalso receive information regarding distance and orientation of objectswithin the room from other types of sensors, such as a distance sensor.In block 822, the processor may select a virtual object for display on ahead mounted display, such as from another computer that is generatingvirtual objects for rendering. In block 824, the processor may receivean input from the user indicating a surface on which to anchor thevirtual object in block 824. The processor may orient the virtual objectin order to anchor it to a first anchor surface in block 826, andgenerate an image of the virtual object on the first head mounteddisplay in block 828. The processor may also update the location andorientation of the virtual object to compensate for movement of the userin block 830, such as to compensate or accommodate movement of theuser's head.

In determination block 832, the processor of the first head mounteddevice may determine whether the content selected or rendered by thefirst head mounted device will be transmitted to a second device. If not(i.e., determination block 832=“No”), the processor may return to block820 to repeat the process in order to generate a continuous display inwhich the virtual object appears to remain anchored to the same surfaceeven as the user moves or turns his/her head.

If the processor determines that the selected content generated by thefirst head mounted device should be shared with a second head mounteddevice (i.e., determination block 832=“Yes”), the processor of the firsthead mounted device may receive a second input from a second userindicating a second anchor surface in optional block 834. In optionalblock 836, the processor may optionally perform calculations todetermine a display position and orientation in order to anchor thevirtual object to the identified second anchor surface. In optionalblock 838, the processor of the first or second head mounted device maydetermine a visual orientation of the second user, such as by usingmethods described herein based on sensor data and obtained images. Asmentioned above, the process of orienting the head mounted device withrespect to the surroundings, and determining distances to and relativeorientation of various surfaces may be accomplished continuously bymonitoring orientation sensors (e.g., accelerometers and gyroscopes) andtriangulating its own position and angle of view as the device moves inspace by constantly referencing shifts in the topography of the scannedthree-dimensional (3D) environment, thus performing SimultaneousLocation and Mapping (SLAM) operations.

In a further embodiment, the processor within the first head mounteddevice may determine the orientation of the second user based on imagesof the second user obtained by its own stereo cameras and usinganatomical models. In optional block 840, the first head mounted devicemay transmit data regarding the virtual object to the second headmounted device in an orientation based on the second user's orientationdetermined in block 838. The second head mounted device may then renderan image of the virtual object on the second head mounted display asdescribed here. This process of determining user orientations andgenerating displays may continue in a loop so that the image continuesto appear anchored to the selected surface while the first and secondusers move and turn their heads.

In various embodiments, either head mounted device may perform thecalculations that enable rendering the virtual object in the properperspective for the second user. For example, the processor of the firsthead mounted device may transmit data to the second head mounted devicethat provides the virtual object data in a manner that correctly orientsthe second virtual object for the second user. The process of orientingthe virtual object by the first head mounted device may be based on thedetermined orientation of the second user that is transmitted from thesecond head mounted device to the first. In other words, the second headmounted device may inform the first head mounted device about itsposition and orientation, and the first head mounted device may use thisinformation to orient the virtual object data. Alternatively, the firsthead mounted device may determine the orientation of the second headmounted device based on images of the second user, and use thisdetermination to orient the virtual object data. For example, theprocessor may determine a proper orientation for the virtual object bycapturing images of the second user over time and applying an anatomicalanalysis to the images to detect a body part within the images. Usingthis information, the processor of the first head mounted device maytransmit data regarding the virtual object to the second head mounteddevice to enable its processor to render an image of the virtual objectpositioned at the second anchor surface consistent with the seconduser's perspective.

The processor may also orient the second virtual object based on thedetected body part and structures in view. For example, a second user'shead, neck and torso may be detected in images captured by the firsthead mounted device, and the processor of the first head mounted devicemay determine that there is a horizontal wall or vertical table close bythe second user that is or has been selected by the second user as theanchor surface. As another example, the processor of the first headmounted device may recognize the second user's head, neck and torso inobtained images, and from the positions of these body parts, determinehow the virtual object should be oriented so the second user can readtext associated with the virtual object. For example, images of thesecond user's head and torso may be analyzed to define horizontal (e.g.,from shoulder to shoulder) and vertical (e.g., from torso to head)coordinate axis that may then be used to determine a proper orientationfor rendering the virtual object. Additionally, a viewing angle of thevirtual object image may be determined, such as in terms of distancesfrom the second user's head and torso of each pixel of the virtualobject as presented on the head mounted display. This processing mayinclude varying object brightness consistent with the view angle ofvarious surfaces on the virtual object from the second user'sperspective.

In another embodiment, the processor may capture images of the seconduser over time and may apply an anatomical analysis to the images todetect a body part within the images. For example, a pair of eyes (orthe head mounted device) of the second user and a nose may be detected.The processor may utilize the body part and determine a location wherethe second user is looking and orient the second virtual object based onwhere the second user is looking. In yet another embodiment, theprocessor may determine from the captured images whether the second useris sitting or standing. The processor may generate the second virtualobject as anchored to a planar surface adjacent to the second user ifstanding. If sitting, the processor may generate the second virtualobject to appear anchored to a nearby surface adjacent to the seconduser.

If the processor determines not to deliver content from the first userto the second user in block 832 (i.e., determination block 832=“No”) theprocessor may display a virtual object for the first user in block 838and may update the virtual object for the movement of the first user inblock 840.

In an embodiment, the processor may communicate the signal correctlyorienting the second virtual object for the second user. The processormay also communicate a signal generating the oriented second virtualobject for the second user. These may be communicated to a communicationnetwork. The communication network may receive the signal and the secondvirtual object may be generated for the second user at a distance awayfrom the first user. For example, the first and the second user may belocated in the same room or in different cities. In yet anotherembodiment, the second user may provide additional inputs by way of adetected gesture or an audio command to move or reorient the virtualobject. The processor may also receive an input from the second userindicating whether to confirm or reject a preferred placement of thesecond virtual object. The second user may also provide an input torotate and reorient the virtual object as desired.

In addition to enabling a shared experience between two or more userswearing head mounted devices, image data generated by one or more of thehead mounted devices involved in a collaboration may be transmitted toand displayed on head mounted devices or other mobile devices such assmartphones or tablet computers used by non-participants, enablingothers to view the collaboration in a virtual or augmented realityexperience. As described above, others viewing a collaboration sessionwearing head mounted devices or using another mobile device such as asmartphone or tablet may not only see the virtual objects and userinteractions with them, but have limited interaction capabilities withthe virtual augmentations seen by one of the head mounted device users.This limited interaction may include touching the augmentation to causean effect, defining an interactive area or anchor point on the physicalsurface (effectively adding a new augmentation to the sharedexperience), and interacting with the shared mixed reality scene viagestural and/or audio inputs. Any such changes in the virtual experiencemay be reflected in the head mounted displays worn by the other users.

FIG. 9A illustrates an embodiment method 900 for rendering images of ananchored virtual object for two users in widely separated locations. Forexample, one user may be located in San Diego, Calif. and a second usermay be located in a second location, such as, for example, Cambridge,United Kingdom. However, in both locations, the displayed anchoredvirtual object 14 remains anchored to the anchor surface in eachlocation fixed to a predetermined area of the image. In someembodiments, one user may select an anchor surface for a virtual objectfor multiple other users. Method 900 illustrated in FIG. 9A may beimplemented in computing devices having a processor configured withprocessor-executable instructions to perform the operations of themethod 900. In block 901 of method 900, the processor may commenceoperation by processing an image using data from distance sensors,camera data and calculated distance data.

The scene sensor 500 described above with reference to FIG. 5A mayinclude a near-infrared (NIR) pulse emitter 503 c and an infrared camera503 b with a fast gating mechanism capable of determining time of flightinformation for reflected infrared pulses. Based on the speed of lightand the timing of infrared emission pulses from the illuminator 503 cand the gating of the infrared camera 503 b, distances to objects withinthe captured image may be determined based on the timing of receivedpulses. The pulse signal may be collected for each pixel or groups ofpixels in the image and used to determine a distance from the distancesensor to the objects in the scene, which may be recorded and stored ina storage medium.

In block 902, the processor may calculate parameters to display theanchored virtual object in the image. The virtual object may be intendedto be fixed to a predetermined surface after an input is received by theprocessor and the calculated parameters may relate to the desiredsurface. In block 903, the processor may render the image for displaywith the virtual object. In block 904, the processor may control atransmitter/receiver or transceiver to upload and/or download a roomcoordinate map to a server. Such a coordinate map may be any form ofposition defining information that enables the head mounted devices todetermine locations of structures within the room within a convenientcoordinate system. For example, a square room may be defined forpurposes of such a room coordinate map in terms of Cartesian coordinatesin which the X and Y dimensions correspond to major axes of the floorand the Z coordinate corresponds to the vertical dimension (i.e., floorto ceiling). Since the cameras and other sensors within the head mounteddevices (and/or body mounted cameras) may locate the major structuraldimensions of a room, such as the walls, floor and ceiling, theorientation of the room coordinates may be easily aligned to the wallsand floor of the room. In this manner, the systems need not be concernedwith geographic coordinates (e.g., from GPS sensors) or compasscoordinates. On the other hand, when users are outside or within a spacethat lacks a convenient Cartesian reference frame, Cartesian coordinatesmay be aligned to GPS and compass coordinates.

With a coordinate frame of reference identified, structures and surfaceswithin view of the users may be identified or linked to the coordinatesystem by using distance measurements from the user (e.g., from distancesensors and/or processing of stereo images) to various coordinatedefining surfaces (e.g., walls, floor and ceiling) and to variousobjects and surfaces in view of the head mounted display. A simplegeometric coordinate translation algorithm may then be used to translateuser-relative coordinates of objects and surfaces to a general roomcoordinate map. Thus, the result of this processing may be to generate adata set of coordinate vectors (e.g., 3 coordinate distances) for eachobject or surface in the room within a general reference frame. A headmounted device may use this coordinate database to determine its ownlocation within the room. Then, knowing its own location within acoordinate database for the room, the processor can calculate distancesand directions from the head mounted device to all objects and surfacesin the room using simple Cartesian coordinate transformation algorithms.In the situation of the users being outdoors, the general referenceframe may be based upon geographic coordinates, which may be arbitrarilyselected and tied to or determined from GPS system measurements.

In another embodiment, a room coordinate map may be stored in a localmemory accessible to the head mounted device, such as in a memorycoupled to a processor of the device, or in a mobile computing device(e.g., a smart phone) wirelessly coupled to the head mounted device. Inblock 916, the virtual object may be rendered on the display on thefirst anchored surface, and the processor may continue to render thevirtual object on the first anchored surface as the user moves and turnsthe head mounted display. In block 917, an audio link may be establishedbetween the first and the second head mounted audio devices forcommunication and collaboration.

Turning to FIG. 9B, in block 905, a processor in the second head mounteddevice, which may be located a geographic distance away from the firstuser, may determine a location and viewing orientation of the seconduser. This may be accomplished using any of the locating methodsdescribed herein, including obtaining coordinate location data, compassdata, and accelerometer data to determine a location and viewing angleof the head mounted device. In some cases the second head mounted devicemay determine its viewing angle by recognizing a part of the user's bodypresent within the image, and from that infer a likely viewingperspective using an anatomical model. In another embodiment, the seconduser may provide an input to designate a surface on which to render avirtual object, and indicate the user's viewing perspective. In block906, the processor may process the virtual object to render the virtualobject appropriately for display from the second user's perspective. Inblock 907, the processor may generate an image in the second headmounted display based on the location and the orientation of the seconduser.

In determination block 908, the processor of the second head mounteddevice may determine whether the second user would like to select a newanchor surface to anchor the virtual object in the displayed image. Forexample, the second user may wish to anchor the image on a surface inthe displayed image that may be convenient during collaboration such asa physical desk top. In an embodiment, the virtual object may beanchored on a virtual physical surface, such as, for example, a virtualtable, which may appear within the rendered image.

If the processor determines that the second user desires to change theanchor surface for the virtual object (e.g., via an input) (i.e.,determination block 908=“Yes”), the processor may determine distance andorientation parameters of the newly designated anchor surface in block909. In block 910, the processor may change the rendering of the virtualobject based on the parameters of the anchor surface (desktop), andprocess the virtual object based on distance and orientation of theanchor surface of the second predetermined surface in block 911. Inblock 912, the second processor may render the image with the anchoredvirtual object on the anchor surface in the display of the second headmounted device.

In block 913, updates of the virtual object received from the userwearing the second head mounted display may be transmitted to the firsthead mounted display for rendering. In this manner, the users of the twohead mounted displays may collaboratively interact with the virtualobject during a collaboration session. These updates may be transmittedbetween the two head mounted devices so each device displays the updatedvirtual image reflecting all changes made by either user. For example,in a word processing application, the virtual object may reflect changesfrom both users in a collaborative manner. In block 914, an audio linkmay be established between the second and the first head mounteddevices. The users may utilize the audio link to speak to one another,as well as other users in an audio conference format. This conferencemay occur at the same time as viewing the virtual object on the displayin two different geographic locations. In an embodiment, the headmounted device may use video and audio conferencing software.

If the processor determines that the second user does not wish to anchorthe virtual object in a new location (i.e., determination block908=“No”), the processor may render the virtual object in block 915. Forexample, the second user may simply select to display the virtual objecton original anchor surface or in a free floating arrangement on the headmounted display. For example, the processor may display the virtualobject in a “stand-by mode” in block 915.

FIG. 10 illustrates an embodiment method 1000 for initiating a peer topeer discovery between a first head mounted device and a second headmounted device. Method 1000 may be implemented in a computing devicehaving a processor configured with processor-executable instructions toperform operations of the method 1000. Once two head mounted devices arelinked through a peer-to-peer communication link, the two head mounteddevices may share resources and may share and exchange data between thedevices. Method 1000 is not limited to linking head mounted devicestogether, and may be applicable to other computing devices that may linkwith one or more head mounted devices. In an embodiment, the headmounted device 10 may be operable as an accessory of a mobile devicehaving a CPU (e.g., phone, tablet, etc.) and a portion of the operationsin method 1000 may be performed by a CPU of the mobile device while thehead mounted device may perform fewer operations, serving as a“terminal” and may receive an output from the mobile device via awireless interface. For example, the processing may be performed on afirst mobile device and a display output may be delivered to the headmounted display which generates the viewable image of the anchoredvirtual object.

In block 1002 of method 1000 the processor may receive position datafrom a locating device associated with the head mounted device, such asa Global Positioning Unit (GPS) or any other space-based globalnavigation satellite system (GNSS) receiver, or navigation deviceassociated with the head mounted device. For example, the locationdevice may be a device that determines a location from a radio networkbase station. In another embodiment, the head mounted device may receivelocal positional data from a local device, such as a cloud server, orfrom a wireless communication network base station. In a furtherembodiment, the user may provide an input (e.g., a voice command orbutton press) indicating the head mounted device is entering a newlocation and/or starting a new collaboration session. In a furtherembodiment, the head mounted device may receive peer networkadvertisements from other devices and determine from a new advertisementthat the device has entered a new location.

In determination block 1004, the processor may determine whether thefirst head mounted device has entered a new location based on positiondata or other indications. If so (i.e., determination block 1004=“Yes”),the head mounted device may initiate peer-to-peer discovery in block1006 to determine if other computing devices or head mounted devices arein the vicinity and can be linked by wireless data connections. Theprocess of initiating peer-to-peer discovery in block 1006 may continueuntil a response is received from a device that can be linked (i.e., solong as determination block 1008=“No”). When a device capable ofestablishing a peer-to-peer communication link is identified (i.e.,determination block 1008=“Yes”), a peer-to-peer communication link maybe negotiated in block 1010, and a two-way communication establishedbetween the two devices in block 1012.

In block 1014, the head mounted device may receive map data from thecomputing device with which it has established a communication link. Inthis operation, the head mounted device may download any map data thathas been stored on the other computing device (e.g., another headmounted device, another computing device or a distant server, etc.) sothat the head mounted device can immediately orient itself within thenew location. For example, if the computing device with which the headmounted device has established a communication link has access to anetwork or the Internet, map data may be downloaded from a remote server(e.g., from the “cloud”). The head mounted device may also scan its owninternal memory to determine whether map data for the new location isalready stored in memory. If map data is stored in memory, as part ofthe operations in block 1014, the head mounted device may share some ofthat map data with the computing device with which the communicationlink has been established. Also, before exchanging map data, thecommunicating head mounted devices may determine whether they are closeby each other, and only exchange map, distance, and orientation data ifthey are close enough together that sharing map data will reduceprocessing burdens on one or both device processors. Thus, if the headmounted device and the linked computing device are not in the same room,it may not make sense to share location, map or orientation data thatwould be irrelevant to the other device.

In block 1016, the processor may receive data from the head mounted orbody mounted stereo cameras, as well as distance measurements that maybe obtained by distance sensors. In block 1018, the processor may useany map data received in block 1014 in conjunction with the cameraimages and any distance measurements in order to orient the devicewithin the map and scene. Optionally, in block 1018, the processor maygenerate or update three-dimensional map data for any items orstructures seen in the images that do not appear within the received mapdata. If no map data was received from another computing device and thedevice did not have a map for the location in memory, the processor maygenerate a new map in block 1018.

In block 1020, the head mounted device processor may process imagesobtained by the camera using the data from the various sensors,including distance sensors and orientation sensors. This processing mayenable the device to locate itself within coordinates or locationswithin the map data. This processing may also include determining thedistance to and orientation of an anchor surface to use in renderingvirtual objects.

In block 1022, the head mounted device processor may exchange displayand virtual object data with the linked computing devices in order toenable the devices to generate the displays that may be used incollaboration. In block 1024, the head mounted device may beginsupporting the user with collaboration, such as by displaying virtualobjects on anchor services, detecting and responding to user commandgestures, and communicating changes and inputs to other coupledcomputing devices as described herein regarding the various embodiments.

FIG. 11 illustrates an embodiment method 1100 for processing recognizedgestures and detected audio as input commands. The gestures and detectedaudio may execute tasks and instructions on the head mounted display.Method 1100 may be implemented in a computing device having a processorconfigured with processor-executable instructions to perform operationsof the method 1100.

In this embodiment method, user input commands received in one form,such as recognized user gestures, may be confirmed by a user input inanother form, such as a verbal command received by a microphone. Forexample, if a processor of the head mounted device recognizes a firstcommand, such as by processing image data to recognize gestures, theprocessor may monitor other user inputs for a short period of time for asecond command that confirms the recognized first command. In thisembodiment, the processor may not implement an action or execute a taskcorresponding to the first recognized user input unless that action ortask is confirmed by a second user input. The second user input may be avoice command received by a microphone, or a second gesture that isrecognized within images gathered by the body mounted cameras. Thisembodiment may be useful since a number of normal user movements may bemisinterpreted as command gestures; requiring the user to confirmgesture commands before they are executed and guard against unintendedcommands being executed. The process of waiting for a confirmatory userinput may be invoked when a gesture made by the user is recognizedwithin an image, or the processor believes it has received a verbalcommand. Generally, this two command confirmation process may avoid asituation where a stray gesture or an inadvertently spoken command mayunintentionally be acted upon by the head mounted device. Instead, thehead mounted device will await a second command before taking an action.This two-command process may also be implemented based upon the type ofcommand that is recognized, such as when the action associated with arecognized command involves an operation that may be undesirable orirreversible, like deleting a piece of content.

In block 1101 of method 1100, the processor may commence operation byreceiving sensor data regarding the orientation of the head mounteddevice. In block 1102, the processor may receive data from the camerasregarding the captured image. In block 1103, the processor may calculatedistance and orientation of objects in the image. In block 1104, theprocessor may process the image using sensor data, camera data and thedistance sensor data. In block 1105, the processor may render the imageon the display with the virtual object contained in the image.

In block 1106, the camera data may be processed for determining whetherany anatomical features are contained within the view of the camera.Generally, an anatomical feature detection algorithm may be applied tocaptured camera image data in order to determine whether any featuresare recognized as being part of the human. For example, the anatomicalrecognition algorithm may scan the images received from the body mountedcamera to determine if an anatomical feature is recognized, such as ahand with five fingers and an arm. The process of recognizing anatomicalfeatures may also determine their orientation, such as the angle of auser's arm within the captured image. Positions and orientations ofrecognized anatomical features may be tracked in images over a timeperiod and compared to a database or dictionary of user gestures todetermine movements of the tracked anatomical features correspond to arecognized gesture. If a user gesture is recognized, the processor mayoutput a command in response to the detected gesture. For example, thecommand may open a software application, or may provide a signal orinput to a software application.

The processor is recognizing user gestures may be accomplished indetermination block 1107 by comparing movements of recognized anatomicalfeatures to a database of predefined anatomical movements, such as maybe maintained in a gesture dictionary. For example, a body part, such asa hand, may be recognized in the captured images and its movementstracked over a predetermined time interval. For example, if theprocessor, using the anatomical model, recognizes that the user's handand the fingers are positioned in front of the body mounted camera withtwo fingers outstretched and three fingers are folded downwardly towarda palm of the user, the processor may determine by comparing thisorientation of the user's hand and fingers to a gesture dictionary thatthe that the user is expressing a desire for the processor to take anaction. The processor may recognize body part gestural arrangements bycomparing image patterns and movements to movement data or patternsstored in a gesture dictionary. Such a gesture dictionary may storegesture data for movements that may include pokes, pats, taps, pushes,guiding, flicks, turning, rotating, grabbing and pulling, two hands withpalms open for panning images, drawing (e.g., finger painting), formingshapes with fingers (e.g., an “OK” sign), and swipes, all of which maybe accomplished on or in close proximity to the apparent location of avirtual object in a generated display. If the head mounted displayrecognizes a gesture in the camera image data that matches an entrywithin the gesture dictionary (i.e., determination block 1107=“Yes”),the processor may process an input command that corresponds to therecognized gesture in block 1108. In block 1109, the input command maybe executed by the processor. The input may also be any signal or datathat is communicated to a software application, or a computing device,or a signal to provide an indication.

Again, as discussed above, the execution of a command based on therecognized gesture may not be accomplished until a second confirminguser input is received. Thus, in such an embodiment, instead ofexecuting the command, the processor may display a prompt (or generate atone) for the user to confirm the recognized gesture command in block1109. In determination block 1110, the processor may determine whetherany audio (or other user input) is received from a microphone for ashort period following recognition of a gesture command. If the audio isreceived within that period of time, the processor may process the audioto determine whether a confirmatory command was spoken to confirm thedetected gesture. To accomplish this, audio inputs received from themicrophone may be digitized and the resulting data compared to a voicecommand recognition database for words that correspond to one or moreinput commands. When the processor determines that an audio inputmatches a voice command pattern corresponding to command confirmation(i.e., determination block 1110=“Yes”), the processor may process aninput command that results in an action that corresponds to the audio inblock 1111. In block 1112, the processor may execute the recognizedcommand and an indication or the recognized task may be displayed on thehead mounted in block 1113.

For example, a user may gesture to anchor the virtual object 14 on aphysical surface by pointing. Upon recognizing this gesture, theprocessor may generate a prompt, such as an audible tone or messagepresented in the head mounted display requesting the user to confirm thecommand. To do so, the user may speak words like “okay,” “confirm” or“make it so” to confirm that a gesture command recognized by the headmounted device should be executed. Thus, when the processor detects theconfirmatory or audible command, the processor may present images in thehead mounted display that shows the virtual object anchored on thephysical surface to which the user is pointing.

If no audio is received that is recognized as corresponding to a spokenaudio command (i.e., determination block 1110=“No”), the processor maydetermine that the gesture that was detected was inadvertent and theprocessor may ignore the recognized gesture and input command and maycontinue to process camera data looking for anatomical gestures in block1107.

Returning to determination block 1107, if the processor does notrecognize a gesture within the image data (i.e., determination block1107=“No”), the processor may continue to process camera data foranatomical gestures for a further additional time interval in block1106. The processor may display a task identifier or icon on the headmounted display in block 1113 indicating that no gestures have beenrecognized. Alternatively, the processor may display a task identifieror icon as soon as a gesture command is been received, which may serveas a prompt for the user to issue a confirmatory command, such as averbal command or an “OK” gesture command.

In another embodiment, different first and second commands may be used.In another embodiment, the processor may be configured to detect anaudio command first and look for a confirmatory gesture command withinreceived images, effectively swapping the operations performed indetermination block 1107 and 1110. In further embodiments, more than onetype of confirmatory input may be required, such as an audible commandin combination with a button press, etc.

FIG. 12 illustrates an embodiment method 1200 for processing recognizedgestures as input commands to execute software tasks on the head mounteddevice. Method 1200 may be implemented in a computing device having aprocessor configured with processor-executable instructions to performthe operations of the method 1200.

In method 1200 in block 1201, the processor may commence operation byreceiving sensor data regarding an orientation of the head mounteddevice 10. In block 1202, the processor may receive stereo image datafrom the stereo cameras, as well as data from any distance sensorsincluded in the head mounted device 10 described above. Thus, in block1202 the processor may obtain all information gathered by the headmounted device regarding images and distances to surfaces within thefield of view of the camera(s). In block 1203, the processor maycalculate distance and orientation data of objects in the capturedimage. These calculations may use well known trigonometric methods whenstereo images are provided in block 1202, direct measurements whendistance sensors are used to provide distance data, and combinations ofdistance information obtained from sensors and calculated from stereoimages. In block 1204, the processor may process the image usingdistance sensor data, camera data and the distance and orientation data.In block 1205, the processor may render the image on the head mounteddisplay with the virtual object. In block 1206, the camera data may betracked by the processor. The captured images may be processed and theprocessor may determine whether an anatomical feature is containedwithin the captured images. The processor may apply an anatomicalfeature detection algorithm to the captured images to determine if anyfeatures or at least one body part is located within the data. Theprocessor may track the body part over a time interval to determine if apredetermined gesture is recognized.

In determination block 1207, the processor may determine whether anygestures are recognized in the captured image. If so (i.e.,determination block 1207=“Yes”) which indicates the processor hasrecognized a gesture in the image, the processor may substantially matchthe determined gesture with an input command stored in memory in block1208. The processor may execute the input command that corresponds tothe detected gesture in block 1209. In block 1210, processor may executea task that corresponds to the input command. If the processor reaches adecision that there are no recognized gestures (i.e., determinationblock 1207=“No”), the processor may continue tracking the capturedimages and applying the detection algorithm to the captured images inblock 1211 to track for an input.

FIG. 13 illustrates an embodiment method 1300 for detecting objects inthe stereoscopic image and providing an interesting visual virtualreality effect based on the detection and interpretation of the detectedobjects. Method 1300 may display “deemed essential objects” on the headmounted display while the method 1300 may render other “unessentialobjects” as transparent on the head mounted display. Method 1300 may beimplemented in a computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 1300. In method 1300, the processor may commence operation byreceiving sensor data regarding orientation of the camera and headmounted device in block 1301. In block 1302, the processor may captureimages and receive data from the cameras. In block 1303, the processormay calculate distance and orientation of objects in the image. In block1304, the processor may process the image using distance sensor data,camera data, orientation, and the distance data.

In block 1305, the process may render the image on the display with thevirtual object 14. In determination block 1306, the processor maydetermine whether any objects are recognized within the images. Forexample, an anatomical detection algorithm may be applied to thedetected images and data to determine if a body part is detected in theimages. For example, an anatomical or a skeletal detection algorithm maybe applied to the captured images to detect a body part, such as, an armor a hand that may be rendered as visible or as transparent. If theprocessor determines that other objects are detected (i.e.,determination block 1306=“Yes”) which indicates the objects aredetected, the processor may determine whether to render the objecttransparent by superimposing a virtual object on the object so as tomake the object appear as if the object has been omitted from the image.

In the various embodiments, virtual objects may be presented with avariety of transparency characteristics. In some embodiments, a virtualobject may be displayed as appearing over an outline of a real worldobject (arm, hands, fingers, and face). In such an embodiment, thevirtual object may be changed in shape to match or cover over a realworld object. In this manner, the real world object appears to be“see-through” in the image output on the head mounted display. In anembodiment, if the unnecessary object is in front of a wall or table, avirtual object may be generated that resembles the wall and/or table andsuperimpose the virtual wall/table over the unnecessary object. If theprocessor determines that no other objects are detected (i.e.,determination block 1306=“No”), the processor may continue and renderthe image with the virtual object in block 1310.

In determination block 1307, the processor may determine whether anyobjects are recognized within the images are necessary. For example,deemed “necessary objects” may be displayed while other deemed“unnecessary objects” may be rendered transparent to omit the objectfrom the image. For example, a deemed necessary object may be a user'sown hands in the image but another object that is deemed unnecessary maybe a second individual's hands located over the virtual object. Theunnecessary object may be displayed in an outline form with the virtualobject superimposed over the unnecessary object. The terms necessary andunnecessary may be arbitrarily programmed and selected by the userutilizing a drop down menu or any supported gesture or method for theuser to indicate a preference, such as gaze detection, a hand pointinggesture that may be recognized by the head mounted or body mountedcameras, etc. The user may program the head mounted device to display“other users' features” as transparent and display “my features always”,or vice versa. If the processor determines that the objects arenecessary (i.e., determination block 1307=“Yes”), the processor mayprovide program instructions to render the image in block 1310 with thenecessary object and the virtual object. For example, a user's own handsmay be deemed necessary and displayed in the image. The terms necessaryand unnecessary are arbitrary parameters and may be determined by theuser. For example, the user at a set up operating procedure may indicatethe objects that are necessary and the objects that are unnecessary andshould be rendered as transparent. For example, the user may determineusing a drop down menu and an input device that the user's hands arenecessary and that other user's body parts in the image are notnecessary. For example, many different users may be present in the imageand collaborating over an anchored virtual object. The scene may becluttered with many users and many virtual objects. Therefore, some ofthe users may be rendered transparent so as to alleviate cluttering inorder to enhance collaboration.

If the processor determines that the objects are unnecessary (i.e.,determination block 1307=“No”), the processor may render the image withthe object as transparent in block 1308. In block 1309, the processormay also display the virtual object with the unnecessary object renderedas transparent or with the virtual object displayed over the unnecessaryobject. In another embodiment, the object may be rendered partiallytransparent only when the object occludes the virtual object 14 and maybe rendered solid when the object does not occlude the virtual object14.

FIG. 14 illustrates an embodiment method 1400 for rendering images on ahead mounted display taking into account hands, arms and other humanfeatures (e.g., those of the user and another individual) that mayappear in the image scene. In this embodiment, the head mounted deviceprocessor may capture an image of a scene in which a virtual object isto be displayed, recognize a body part present in the captured image,such as by processing the image using an anatomical model, and adjustthe display of the virtual object based upon the recognized body part.The adjustment to the display may involve rendering the virtual objectin front of, behind or blended with a recognized body part. For example,the processor may render a virtual object as partially transparent inplaces where a user's feature (e.g., hands and arms) occludes thevirtual object and as nontransparent where the user's feature does notocclude the virtual object. In this manner, the user can see his/herhands in the vicinity of the image of a virtual object while the rest ofthe virtual object appears solid as if real. Alternatively, theprocessor may render the virtual object so the user's own featuresappear in front of or over a virtual object (i.e., solid) while thehands and arms (and other body parts) of another individual are renderedtransparent so they are made invisible by the image of the virtualobject (i.e., the virtual object blocks the user's view of the otherperson's hands/arms), or so that the virtual object can be viewedthrough the other person's hands and arms.

For example, the head mounted or body mounted camera may capture imagesof the scene which include hands and arms of the user and anotherindividual. The device's processor may apply an anatomical algorithm tothe captured image data to recognize body parts within the capturedimages. For example, the processor may detect a user's hands and asecond individual's hands in the captured images by comparing the imageshapes to an anatomical model.

Both sets of hands may occlude a virtual object. For example, fivefingers and hand size data may be reviewed for the image. The processormay infer that hands which are oriented from top down on captured imagesare the second individual's hands while other hands that are orientedfrom bottom up within captured images are the user's hands.Additionally, the processor may capture distance data of the recognizedobject to formulate an inference. For example, when the processordetermines that anatomical features (e.g., arm and hands) are at least apredetermined distance (twelve feet) away from the user, the processormay infer from this information that those features belong to anotherindividual. Additional sensed parameters are also possible and withinthe scope of the present disclosure. For example, the height of theobjects, the orientation of the objects, and the size of the objects mayall be utilized.

In method 1400, the processor may commence operation by receiving sensordata regarding the orientation of the cameras block 1401. In block 1402,the processor may receive data from the cameras. In block 1403, theprocessor may calculate a distance and an orientation of objects in theimage. In block 1404, the processor may process the captured image usingdistance sensor data, camera data, orientation of the image and thedistance data. In determination block 1405, the processor may determinewhether an object is recognized within the images that corresponds to auser's feature, for example, the user's hands, arms, legs, torso, etc.This determination may involve applying an anatomical analysis to theimage by recognizing objects within the captured image, comparing therecognized objects to a skeletal model stored in memory, and determiningif any of the recognized objects match a portion of the skeletal modelin response to comparing the recognized objects to the skeletal model.The skeletal model may include relative position information about keyjoints, bones, and limbs of a human skeleton.

If a feature of a person is recognized within the image (i.e.,determination block 1405=“Yes”), in block 1409, the processor may renderthe image with the virtual object and the user's features in the image.If no user's features are detected within the images (i.e.,determination block 1405=“No”), the processor may determine whether thefeature is the second individual's features in determination block 1406.

In determination block 1406, the processor may determine whether thesecond individual's features are detected. This determination mayinvolve determining whether the body part belongs to a first individualor to a second individual in response to recognizing a body part presentin the captured image, determining a location and encompassed area ofthe virtual object, and determining whether the recognized body partoverlaps at least one portion of the area encompassed by the virtualobject. These features may be rendered transparent by the processor. Theprocessor may omit the object(s) by generating and superimposing avirtual object on the detected feature. For example, the processor mayrender the virtual object over the second individual's hands on thedisplay. If so (i.e., determination block 1406=“Yes”) which indicatesthe second individual's hands are detected and are not necessary in theimage, the processor may render the virtual object superimposed in theimage over the second individual's hands in block 1407.

If the processor is unable to determine whether the object is the secondindividual's hands (i.e., determination block 1406=“No”), the processormay render the image with the virtual object in block 1408. Theprocessor may determine a number of pixels in the image that correspondto a desired object and then leave an outline of the desired object asdisplayed, but replace pixels in the image with a virtual object 14. Theoutline size may vary and may include about 1 percent of the objectimage length.

FIG. 15 illustrates a high level top down view of an illustration of adisplay output of the head mounted device 10. The display output shows avirtual object 14. The virtual object 14 is illustrated as a virtualflat panel screen that outputs data. In this embodiment, the virtualscreen outputs a graphical table with a number of bar charts. Multipleusers may collaborate over the virtual object in a virtual realitysetting in that the virtual object 14 only appears in the output of thedisplay but appears to be a tangible, real object to the viewer.

In the illustrated example, the processor detects a user's features orhands 1502 and detects two other individuals' features or hands 1500 and1501 over the virtual object 14 within the stereoscopic image. Asillustrated in FIG. 15, the user's hands are allowed to be displayed bythe processor as an essential element in the display. The user's handswill appear over and will occlude the virtual object 14. Additionally,two other individuals' features may be rendered seemingly transparent onportions of the virtual object 14 since these elements are deemed to benot essential and may distract or may clutter the view of the user ofthe head mounted device 10. The virtual object 14 may be displayed bythe processor as superimposed and fitted over the hands 1500 and 1501 sothe hands 1500 and 1501 do not occlude the generated virtual object 14.

In the illustrated example, the second user's hands 1501 are rendered bythe processor as “transparent” and as shown by the outline of the secondindividual's hands 1501 rendered over the virtual object 14. The virtualobject 14 portion that should normally be occluded may be rendereddirectly on the second individual's hands 1501. In the illustratedexample, the second user's hands 1501 appear to be transparent with onlyan outline of the hands 1501 shown. The other user's hands 1502 arerendered as a “solid object” in the illustrated example, with thosehands 1502 rendered over and occluding the virtual object 14. In theillustrated example, the hands 1502 appear to be solid and the virtualobject 14 may be not visible underneath. In an embodiment, the otherindividual's hands may be detected and rendered with an outline of thehands 1500 and 1501. A remainder of the hands 1500 and 1501 areillustrated as being rendered beneath the virtual object 14.

The head mounted or body mounted camera may capture images and apply ananatomical model to the captured image data to recognize a body part.For example, this may be a user's hands 1502 and a second individual'shands 1500 and a third individual's hands 1501 over the virtual object14. For example, five fingers, hand size data, a portion of the arm, theorientation of the hands, the orientation of the fingers, distance dataand orientation data may be reviewed. The processor may infer that handsthat are oriented from the top of the image and extending toward thebottom of the image are the second individual's hands 1500. Theprocessor may also infer that hands oriented from a right side of theimage and a certain distance away from the camera 1501 are the otherindividual's hands 1501. The processor may also infer that a third setof hands 1502 which are oriented so they extend from the bottom of theimage up are the user's hands if they are within a distance range andpositioned in the image with respect to the orientation of the headmounted device 10 consistent with an anatomical model. Additional sensedparameters may also be used by the processor in recognizing body parts.For example, when the user's hands or arms have been tracked, theprocessor may infer that all other hands in the displayed image belongto other users, and therefore the processor may render those remaininghands as outlines or transparent in the image presented on the headmounted display.

FIG. 16 illustrates another embodiment of the head mounted device 1600communicating with a mobile device 1605 across a high speed data link toleverage processing resources. FIG. 16 illustrates a number ofsubcomponents of the head mounted device 1600. The head mounted device1600 may be operable as an accessory to a mobile CPU (phone, tabletetc.) with the main processing performed on the mobile CPU. Generally,the head mounted device 1600 may be made as lightweight as possible tobe comfortably worn around the head of the user. The head mounted device1600 may leverage the processing and data storage resources of a mobiledevice 1605 and a VIRT control system 1610 may be implemented within thesoftware applications running on the mobile device 1605. The mobiledevice 1605 may provide processing functions, increase memory, andconserve battery life of the head mounted device 1600 by performing theprocessing at the mobile device 1605.

The head mounted device 1600 comprises a scene sensor (a RGB-D camera)1615 connected to a processor. The head mounted device 1600 may alsoinclude an audio sensor (microphone) 1620 and a stereoscopic display1625. The display 1625 may be a semitransparent OLED video display 1625.The head mounted device 1600 may also include an ear bud that includes aspeaker or may alternatively include a pair of headphones 1630. Inanother embodiment, the head mounted device 1600 may comprise a strippeddown device 1600 to be lightweight. The device 1600 may comprise awireless interface, a scene sensor 1615, a battery and a display 1625.The wireless interface may communicate data to a processor in the mobiledevice 1605 which may be configured to run a VIRT control system 1610 sothe head mounted apparatus remains lightweight and comfortable on theuser. As described in more detail below with reference to FIG. 41, amobile device 1605 includes a processor and a memory, and in anembodiment the processor may be configured with a VIRT control softwaremodule 1610 that may be configured to interoperate with the head mounteddevice 1600. The mobile device 1605 and the head mounted device 1600 maycommunicate using a high speed data link 1612, which may be wired orwireless. For example, the high speed data link 1612 may be a shortrange wireless data link 1612 selected from any one or more ofBluetooth®, Wi-Fi®, Qualcomm's Peanut®, and ZigBee® wireless protocols.

For example, sensor data may be captured utilizing sensors 1615, 1620and 1630 and converted to digital data. The digital data may becommunicated to a wireless device and the digital data may becommunicated along link to the control system 1610 operable on themobile device 1605. The control system 1610 may process the sensed dataand may render display data. For example, the control system 1610 mayreceive scanned images and distance data of the area and generate avirtual object on at least one surface in the room. The display data maybe communicated back along the link and output on the head mounteddisplay 1625.

FIG. 17 illustrates an embodiment method 1700 for using a high speedlink between the mobile device and a head mounted device to enable offloading of sensor data processing to the mobile device. Method 1700 maybe implemented in a computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 1700. In block 1701, the processor may receive sensor dataregarding orientation of the head mounted device and its cameras. Inblock 1702, the processor may receive image data from the device'scameras. In determination block 1704, the processor may determinewhether a mobile device is present, wirelessly communicating with thehead mounted device and able to assist with processing and memorystorage. If no (i.e., determination block=“No”), in block 1710, theimages may be processed by the head mounted device as described in otherembodiments.

If a mobile device is present, wirelessly communicating with the headmounted device and able to assist with processing and memory storage(i.e., determination block=“Yes”), the head mounted device may sendsensor data to the mobile device for processing by the mobile deviceprocessor in block 1705. In block 1706, the mobile device processor maycalculate distance and orientation data for objects within the imagebased on the received data. In block 1707, the mobile device processormay process the image data using sensor and camera data received fromthe head mounted device. In block 1708, the mobile device processor maygenerate display data for the virtual object using the sensor and imagedata received from the head mounted device, and transmit the generateddisplay data to the head mounted device. In block 1709, that headmounted device may use display data received from the mobile device torender images on the head mounted display. This process may continue byreturning to block 1701, although once the head mounted device isdetermined that a mobile device is present (i.e., determination block1704=“Yes”), then a determination may not be repeated for a currentsession.

FIG. 18 illustrates a method 1800 for communicating data to a shareduser memory so a second head mounted device may access shared data.Method 1800 may alleviate a processing time to render an image or mayalleviate processing time to generate a virtual object on a properlyoriented display. For example, the processor may, in some embodiments,process image data and also receive image data, or location data from astorage medium to render a virtual object faster. Method 1800 may beimplemented in a computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 1800.

In method 1800, the processor may commence operation by receiving sensordata regarding orientation of the cameras. In block 1802, the images maybe captured and the processor may receive stereo image data from thecameras. In block 1803, the processor may calculate distance andorientation of objects in the image. In block 1804, the processor mayprocess image data from the sensors. In block 1805, the image may berendered with a virtual object. In block 1806, a determination may bemade regarding whether to share data to a remote storage medium soanother head mounted device may receive the shared data and utilize theshared data. For example, the other head mounted device may obtainposition data, or may use the shared data to render the image and thevirtual object.

If the processor determines to share data to a remote storage medium(i.e., determination 1806=“Yes”), the data may be communicated to ashared memory in block 1807. If the processor determines not to sharedata to a remote storage medium (i.e., determination block 1812=“No”),the processor may continue to block 1801 to continue to capture data. Asecond head mounted device 10 b may access the data from the storagemedium in block 1808 and may receive the data in block 1809. In block1810, the distance data may be calculated and the image processed inblock 1811. A determination may be reached in determination block 1812to share data that is aggregated by the second head mounted device 10 b.For example, the data may be scanned sensor data or other data tocollaborate. The data may be transmitted by the processor and stored atthe storage medium. If the processor decides to share data aggregated bythe second head mounted device (i.e., determination block 1812=“Yes”),data may be communicated to a shared memory in block 1813. If theprocessor decides not to share data aggregated by the second headmounted device (i.e., determination block 1812=“No”), the processor maycontinue to block 1809 to continued to receive data regarding the image.

FIG. 19 illustrates an embodiment method 1900 for sharing data over acommunication network and for use by multiple devices. A head mounteddevice 10 may share resources with a second mobile communication device10 b. The method 1900 may be implemented in a computing device having aprocessor configured with processor-executable instructions to performthe operations of the method 1900.

In method 1900, the processor may commence operation by receiving sensordata regarding orientation of the cameras. In block 1902, the processormay receive stereo image data from the cameras. In block 1903, theprocessor may calculate distance and orientation of objects in theimage. The processor may convert the image data and the position andorientation data. The data may be tagged with position data for use by asecond head mounted device. The second head mounted device may receivethe distance, image, and the tagged positional data. The second headmounted device may utilize the data in order to render the image withthe virtual object in the image with minimal processing.

In block 1904, the processor may tag objects in the image to store thetagged information in a geometric three-dimensional model. The tag maycomprise geographic position data that may be obtained from a GPSdevice. In an embodiment, the position data may describe objects in theroom and the relative position of objects relative to one another. Inanother embodiment, each object may be assigned GPS coordinatesincluding an orientation of the objects. In an embodiment, the tag maycomprise at least four separated positional coordinates. In a furtherembodiment, data associated with the virtual object may also be taggedwith position data. In block 1905, the processor may process the imageusing sensor data, camera data, orientation of the image and thedistance data. In block 1906, the processor may render the physicalenvironment in a model. The model may be uploaded and transmitted by theprocessor to a storage medium, which may be accessible by other headmounted device(s) or separate processors connected to other head mounteddevice(s). As mentioned herein, the head mounted devices may beconfigured to be light weight by leveraging external processors andstorage media (e.g., a smart phone, personal computer, or serveraccessible via a local wireless network). A second head mounted devicemay determine its position and may form a request to a storage medium toaccess scanned data relating to a current position. The storage mediummay receive the request. The storage medium may communicate the data tothe head mounted device based on the position.

In determination block 1907, the processor may determine whether toshare the data. If so (i.e., determination block 1907=“Yes”), theprocessor may share the data and communicate the data to a mobile device(block 1908) associated with a second user, or to a storage medium,where a computing device may request data from the storage medium. Forexample, the storage medium may comprise another computing device, aserver, or a network distributed computing device. If the processordetermines not to share the data (i.e., determination block 1907=“No”),the processor may continue to block 1901 to continue to receive datafrom the sensors regarding orientation of the camera and capture imagesin block 1902. In block 1908, a second head mounted device may accessthe shared data. The processor may utilize the shared data to displaythe virtual object in block 1909 while reducing the amount of processingto render the virtual object.

FIG. 20 illustrates an embodiment method 2000 for using a server toshare location coordinate data and scanned data between at least twohead mounted devices to render a virtual object. For example, aprocessor may share data over a communication network for use bymultiple devices so the devices may process image data, positional data,orientation data, distance data and more quickly render a virtual objectin an image by obtaining shared data or may share other data useful forcollaboration purposes between software applications. For example, oneuser may be able to access another user's files stored in memory. Themethod 2000 enables a processor to collaborate and share resources withother devices. Method 2000 may be implemented in a computing devicehaving a processor configured with processor-executable instructions toperform the operations of the method 2000. In block 2001, the processormay commence operation by executing an application associated with afirst head mounted device.

In block 2002, the processor may open an input/output channel that isassociated with the head mounted device. The processor may execute anapplication with second head mounted device in block 2003 and open thechannel in block 2004 to communicate over the channel. The processor mayexecute an application on a shared server in block 2005. In block 2007,the processor may create a connection to the server and both headmounted devices using the RF channel. The processor may collaborateusing the first and the second head mounted devices and the server inblock 2008. In block 2009, the processor may share audio data. In block2010, the processor may share data files and synchronize documents atthe server. In block 2011, the processor may detect audio gestures atthe mobile devices for new commands.

Also, as described above, sharing of image and audio data generated byone or more of the head mounted devices with a server may enable othersto observe the collaborations in a virtual or augmented realityexperience by seeing and hearing the same thing as participants bywearing a head mounted device that receives image and audio data fromthe server. By enabling bi-directional communications between observerhead mounted devices, and between the server and participant headmounted devices, the observers wearing head mounted devices may alsohave the ability to interact with virtual objects seen in the headmounted displays. Observer interactions with virtual objects may includetouching the augmentation to cause an effect, defining an interactivearea or anchor point on the physical surface (effectively adding a newaugmentation to the shared experience), and interacting with the sharedmixed reality scene via gestural and/or audio inputs. Any of suchchanges in the virtual experience made by observers may be reflected inthe head mounted displays worn by the participants and other observers.

FIG. 21 illustrates an embodiment method 2100 using a server to sharedata between at least two devices via an identifiable directory. Thedirectory may identify other capable devices and head mounted devices ina predetermined area. Once a head mounted device enters an area, thehead mounted device may register and may store address information on adirectory, which may be shared with other head mounted devices. Inalternative embodiments or in a possible implementation of theembodiments the scanning function may be performed by an attachedprocessor unit, such as a smartphone. The use of a scanning of an areaby a separate device may be particularly appropriate when a lightweighthead mounted device is used. This way a head mounted device may quicklydetermine from a centralized location if other head mounted devices areavailable for collaboration and to share scanning resources. Method 2100may be implemented in a computing device having a processor configuredwith processor-executable instructions to perform the operations of themethod 2100. The directory may be computer readable data that providesinformation about available devices. This facilitates sharing data overa communication network for use by multiple devices. For example, thedirectory may include information about scanned areas in a building withgeotags associated with the directory so a device 10 knowing itsposition may check the directory to obtain scanned data to render avirtual object 14. For example, the data may include data about thevirtual object 14, may include data about one or more surfaces in thearea, orientation data, object position data, or any other useful orprocessor intensive data, which can be obtained quickly.

Embodiment method 2100 enables collaboration and sharing resources tominimize an amount of processing performed by the head mounted devicesthemselves. In method 2100, the processor may commence operation byreceiving an input request to collaborate from a first head mounteddevice that may be running an application in block 2101. In block 2102,the processor may initiate a peer to peer search for near devices forcollaboration. In determination block 2103, the processor may determinewhether to collaborate with discovered devices. If so (i.e.,determination block 2103=“Yes”), the processor may create a connectionbetween the devices. The processor may collaborate using a two waycommunication link. The communication link may be formed between thefirst and the second head mounted devices 10 and 10 b in block 2104.

The processor may access a directory in a server. Processor utilizingthe directory may determine if other users are available forcollaboration in block 2105 by scanning the directory. In determinationblock 2106, the processor may determine whether to select a new user.The processor may determine whether to select a new user based on aninput or based on a decision rules engine that may collaborate based onone or more predefined relationship rules. If the processor determinesto select a new user (i.e., determination block 2106=“Yes”), theprocessor may proceed to determination block 2107 to determine whetherto create a connection to the new user for collaborating. Indetermination block 2107, the processor may determine whether tocollaborate with the new user. If the processor determines tocollaborate with the new user (i.e., determination block 2107=“Yes”),the processor may create a connection to relevant devices and maycollaborate using the first and the second head mounted devices in block2108.

If the processor determines not to collaborate with the new user (i.e.,determination block 2106=“No”), the processor will output a message tothe user that no further users are expected and to commencecollaboration in block 2109 with the existing participants. If theprocessor determines not to select a new user (i.e., determination block2107=“No”), the processor will assume that the collaboration isdeclined. The processor may output a message that no further users areexpected and to commence collaboration in block 2109 with the existingparticipants.

FIG. 22 shows a high level illustration of a first head mounted device2205 and a second head mounted device 2210 being located at twodifferent geographic locations. FIG. 22 illustrates a number ofcomponents of the first and the second head mounted devices 2205, 2210and also a VIRT service component 2265 that may provide data exchangefunctions between the head mounted devices 2205, 2210. The first and thesecond head mounted devices 2205 and 2210 may share digital assetsincluding a three-dimensional map of a scanned area to reduce processingat the new head mounted device entering the new location. The sharingmay increase a battery life and may assist with rendering images in arapid manner. In another embodiment, the scanned three-dimensional mapof one user may supplement the map of a second user. In anotherembodiment, the map of one user may replace the map of a second user. Ina further embodiment, the map of one user may be referenced and used tocheck another's scanned data. For example, this reference may occurwithin a predetermined time frame.

At the expiration of the time frame or in the event processingconstraints are burdensome the second user may utilize the first user'smap. In another embodiment, the first head mounted device 2205 and thesecond head mounted device 2210 may be located at two far geographiclocations relative to one another and may not scan similar areas.Therefore, while digital assets and virtual objects would be sharedbetween the head mounted devices 2205 and 2210, there may be no need toshare map data as such information would not be relevant tocollaboration. Instead, the sharing of three-dimensional maps may bemost useful for user's that are located close by or in the same buildingwhere the head mounted devices 2205 and 2210 are required to scan anarea.

The first user operates the first head mounted display device 2205 thatmay have a display 2215, a speaker 2220, an image scene sensor 2225 andan audio sensor 2230. The first head mounted device 2205 may alsoinclude a wireless or wired interface 2260 and a controller that includea number of control components. In another embodiment, the head mounteddevice may comprise a display 2215, a wireless interface 2260, a battery(not shown), a speaker 2220, and a microphone/audio sensor 2230.Components include a scene manager block 2235, a user control block2240, a surface manager block 2245, an audio manager block 2250 and aninformation access block 2255.

The second user also operates the second head mounted device 2210, whichinclude similar components. Second device 2210 may have a display 2215a, a speaker 2220 a, an image scene sensor 2225 a and an audio sensor2230 a. The second device 2210 may also include a wireless or wiredinterface 2240 a and a controller. Controller include a number ofcontrol components including a scene manager 2235 a, a user controlblock 2240 a, a surface manager block 2245 a, an audio manager block2250 a and an information access block 2255 a.

Each of the devices 2205 and 2210 communicate with a control provider2265 that may provide service functionality for each of the devices 2205and 2210. The control provider 2265 may assist the two devices 2205 and2210 and may provide services on behalf of the devices so the devices2205 and 2210 remain lightweight and do not need to necessarily performcomputational intensive processing locally. Like any two Internetconnected devices, the control provider 2265 may have to implementfirewalls 2285 and 2285 a to protect them from hacking attacks andmalware (e.g., may be provided by local software, Internet serviceproviders and/or enterprise service providers). Firewalls 2285 and 2285a restrict access to the control provider 2265. In another embodiment,the firewalls 2285 and 2285 a are optional and are absent.

The control provider 2265 may include a directory 2280, which maintainsdata pertaining to many devices including identification and routinginformation to the devices 2205 and 2210. The control provider 2265 mayalso include a scene storage block 2270. The scene storage block 2270processes data relating to surface and contour information of scannedimages. The surface and contour data may include distance, shape andorientation data. The scene storage block 2270 may permit a first userto map a first area, and store the data for the benefit of a seconduser's head mounted device to more rapidly render the scene or to verifyscanned data parameters. The control provider 2265 facilitates sharingdigital assets. The control provider 2265 may provide a communicationlink between the wireless interface 2260 and 2260 a, which may beoptimized. The control provider 2265 may also facilitate connectivity bydifferent network topologies, firewalls and IP addressing architectures.

Each device 2205 and 2210 may embed location data into the datacollected pertaining to the mapped and scanned image and scannedsurfaces. For example, a first head mounted device 2205 may scan a roomincluding objects in the room. The scanning may result in theaggregation and collection of positional data for the objects, contourdata of the objects, distance data from the object to the head mounteddevice, and orientation data of the head mounted device and the objects.The scanning may result in data, which may be communicated to the router2275. Together with the data, the first head mounted device 2205processor may embed location data into the scanned data. The locationdata may be coordinates in longitude, latitude and elevation. The firstcommunication device 2205 designates each surface in a local physicalenvironment. The processor may be able to identify which shared data maybe placed on each surface so another device may quickly be able tolocated and utilize the data.

The first head mounted device 2205 may write data to a cloud basedservice 2265. Such a service 2265 may provide computation, software,data access, and storage services for the head mounted devices. The headmounted devices do not require end-user knowledge of the physicallocation and configuration of the service 2265. The cloud-based service2265 may have a lookup directory 2280 stored thereon. Service 2265resides on a server. The service 2265 resources may be dynamicallyprovisioned to head mounted devices on a fine-grained, self-servicebasis over the Internet, via web applications/web services executable onthe head mounted devices. The service 2265 may be located on an off-sitethird-party provider. The directory 2280 assists the first device 2205and facilitates connectivity via a router 2275 between the remote users2205, 2210. In this manner, the second head mounted device 2210 maydetermine that the head mounted device 2210 is in a particular locationand may communicate the location to the router 2275. The service 2265may involve provisioning of dynamically scalable resources. Head mounteddevices may be loaded with a web-based tool or an application that auser can access and use through a web browser as if the program wasinstalled locally on their own head mounted device. The service 2265 maydeliver applications via the internet, which are accessed from webbrowsers, while the software and data are stored on servers at a remotelocation. Service 2265 may be transmitted through shared data-centersand appearing as a single point of access for head mounted devices. Inresponse, data from the scene store 2270 that was generated from thefirst head mounted device 2205 may be transmitted to the scene managerand surface manager blocks 2235 a and 2245 a to avoid additionalcomputational processing to render the image. The second head mounteddevice 2210 may verify data generated by the scene sensor block 2225 aor may utilize the data transmitted from the scene store block 2270 torender an image on the display 2215 a to avoid additional computationalprocessing to render the image.

FIG. 23 illustrates a high level diagram for a first head mounted device2205. A first head mounted device 2205 may be located in close proximityto a second head mounted device 2210. The head mounted devices 2205 and2210 may exchange data via one or more fast wireless data links. Thefirst head mounted display device 2205 may have a display 2215, aspeaker 2220, an image scene sensor 2225 and an audio sensor 2230. Thefirst head mounted device 2205 may also include a wired interfaceconnection to a user data store 2301, where data may be written foraccess. The first device 2205 may also include a wired or wirelessconnection to shared data storage 2302, which may be a memory that thesecond mobile communication device may access. First device 2205 mayalso have a controller that include a number of control componentsinclude a scene manager 2235, a user control block 2240, a surfacemanager block 2245, an audio manager block 2250 and an informationaccess block 2255.

The second user also operates the second head mounted device 2210, whichinclude similar components to the first head mounted device 2205. Seconddevice 2210 may have a display 2215 a, a speaker 2220 a, an image scenesensor 2225 a and an audio sensor 2230 a. The second device 2210 mayalso include a wireless or wired interface. The second device 2210 maywrite and read data to/from a user data store memory 2303. The seconddevice 2210 may also read and write data to a shared data store memory2302 accessible by the first device 2205. The second device 2210 mayalso have a controller that includes a number of control components ormodules. The second device 2210 may include a scene manager block 2235a, a user control block 2240 a, a surface manager block 2245 a, an audiomanager block 2250 a, and an information access block 2255 a.

In this regard, users may share a set of scanned surface data that havebeen designated. Users may interact with virtual assets that are placedonto the surfaces as virtual objects. The first and second device 2205and 2210 may communicate over a local fast wireless data carrier (forexample Wi-Fi®, Qualcomm® Peanut® short range communication standard,and Qualcomm® FlashLinQ®). If additional users are present, the devices2205, 2210 may further communicate via peer-to-peer connections or abroadcast, unicast, or multicast.

FIG. 24 illustrates an embodiment method 2400 using a server to tag datawith location information and share the tagged data between at least twohead mounted devices via a directory. Method 2400 may be implemented ina computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 2400. The directory may provide information about scannedthree-dimensional maps. For example, the head mounted device may emitinfrared light and calculate a time of flight of the infrared light viaa fast gating mechanism and may store the calculations to determine thedistance certain objects are from the head mounted device 10.Additionally, the head mounted device 10 may scan an area to determinean orientation of the objects, to determine surfaces, and structuralinformation of the objects and height, width, length and distanceinformation.

In block 2401 of method 2400, the processor may commence operation byreceiving data from a head mounted device. In block 2402, the processormay receive image data. In block 2403, the processor may calculate thedistance and orientation. The processor may process the content fordisplay in block 2404. The processor may render the image with thevirtual object contained within the displayed image in block 2405. Indetermination block 2406, the processor may determine whether to tag theimage with location information so that another head mounted device mayutilize the image data.

A second device may include a second processor that may calculate imagedata and may utilize the first device scanning capabilities as aresource to verify the scanned parameters as a reference. In anotherembodiment, the second device may import the scanned data. Using theimported data, the second device may render the displayed image with avirtual object anchored on at least one surface in the displayed image.This may occur when the second head mounted device 10 b enters alocation. The second head mounted device 10 b may determine its locationin an area, for example, a room and then communicate the location datathereby forming a requested for the first head mounted device's 10scanned data.

If the processor decides to tag the image with location information(i.e., determination block 2406=“Yes”), the processor may tag thescanned data with the location information for the directory and storethe data on a server in block 2407 so another head mounted device 10 bmay access the scanned data. In an alternative, instead of storing thedata on a server, in block 2407 the data may be stored in a sharedmemory, such as a memory accessible via a shared network. If theprocessor decides not to tag the image with location information (i.e.,determination block 2406=“No”), the processor may determine that noother devices are available to utilize the shared data and will continuescanning the area in block 2408 to render the virtual object for theinstant user.

FIG. 25A illustrates an embodiment of a system 2500 with three mobiledevices 2501, 2502, and 2503 and a server 2504. The devices 2501-2503and the server 2054 exchange scanned data and map information. The mapmay include scanned parameters of a room. The map may include anidentification of surfaces in the image, such as a flat screen display2530 and a table 2532, including shapes and sizes of objects in theimage. The map may also include a distance and angle that the objectsare located from the device, or coordinate locations within a frame ofreference linked to room or arbitrary coordinates. For example, the mapmay identify locations of objects in terms of a number of meters from anorigin along three perpendicular coordinate axes, such as along thethree axes (e.g., length, width, height) defining a particular corner ofthe room. The map may include a header that relates to athree-dimensional position and location of the device where the objectswere captured. Map data may be uploaded to a server 2504 from eachposition where the device scans image data, and the server may usedistance measurements and images from all head mounted devices and fromall locations to build a map based on the frame of reference linked tothe room. To do so, the server 2504 may perform a coordinatetransformation on all received location data to transform distancemeasurements from head-mounted device-centric spherical coordinates(i.e., distance and angle from the head mounted device to each object)to an arbitrary rectilinear coordinate system, such as to locate objectswith respect to room length, width and height coordinates. Themathematics and algorithms for accomplishing such coordinatetransformations are well known in the art.

The embodiment of a system 2500 is not limited to a head mounted devicebut instead may further be applicable to a different computing device.Alternatively, the head mounted device 10 may be an accessory to amobile CPU (phone, tablet etc.) with the main processing performed onthe mobile CPU and data being exchanged by and between the mobile phoneand a head mounted device (not shown). FIG. 25A illustrates howcollaboration may occur to scan and to render the image quickly using acentralized database server 2504. For example, the devices 2501, 2502,and 2503 may include a device for emitting and sensing light orultrasound for measuring distances to objects and surfaces in a room viatime of flight calculations. In another embodiment, a RGB-D camera maycapture image data. The devices 2501, 2502, and 2503 may utilize thescanned data to generate a virtual object and anchor the virtual objecton at least one surface of the image and render the virtual objecttogether with the captured image to provide a virtual or augmentedreality experience.

The first device 2501 may create a partial map and upload the map to aserver 2504. The partial map may include data pertaining to a first anda second object in the image for location 52.23.0.14. The map mayinclude a header that identifies the location. The map may also includesubstantial data about the scanned images including distance from thedevice 2501, object length, object width and object height, orientationof the objects, and distance that the objects are from a wall, ceiling,doorway, etc. The individual coordinates and positional data of eachobject may also be recorded and stored in the map.

The second device 2502 may report its location to the server 2504 viacoordinates (52.23.0.14), which correspond to a predetermined location,which may have already been scanned by the first device 2501. The seconddevice 2502 may upload the location data to the server 2504. The server2504 may receive the location data. The second device 2502 may receive amessage that a partial map is available. The second device 2502 maydownload a partial map formed by the first device 2501. Substantiallysimultaneously, the second device 2502 may be scanning a differentportion of the room. For example, the second device 2502 may scan aportion of the room adjacent a first table. The second device 2502 mayutilize the scanned data and merge the scanned data with the downloadedmap to render a displayed image with the virtual object. The seconddevice 2502 may also upload the updated map to the server 2504.

The third device 2503 may also provide its location data 52.23.0.14 inthe form of a request. The server 2504 may receive the location data52.23.0.14 and coordinates and select an appropriate data set andtransmit the data set of scanned data in the form of a map to the thirddevice 2503. The third device 2503 may download the map data thatinclude the first device's 2501 scanned data and the second device's2502 scanned data, which may supplement the first device's 2501 scanneddata. In another embodiment, the data may not supplement and instead thesecond device 2502 may scan to determine if the first data set isaccurate and may verify the data set. In another embodiment, if the datais incorrect, the second device 2502 may replace the scanned data of thefirst device 2501.

The third device 2503 may utilize the map to render a displayed imagewith a virtual object. As shown, the third device 2503 may quicklydetermines a location of objects in the image including a flat screentelevision 2530 and a table 2532 utilizing the updated map. The thirddevice 2503 may further capture image data and may scan a new area ofthe room and may also upload data to the server 2504 corresponding to anupdate of the map. In this manner, each of the devices 2501, 2502 and2503 does not need to process the entire image locally and may obtainservices to more quickly render the virtual object and displayed image.

Each of the devices 2501, 2502 and 2503 and server 2504 may be optimizedwith one another to quickly receive the map and piece together thescanned data and map components to render the scene quickly. Forexample, if a device 2501 is in a new location and there have been noscanned images recorded, the device 2501 may scan an entire area. Forexample, the device 2501 may alternatively determine that the scanneddata is old and was recorded some time ago. The device 2501 maydetermine that the scanned data is stale, and decide to scan an entirearea.

Alternatively, the device 2501 may compare the age of the scanned datato a threshold value and may make a determination whether to scan theentire area. For example the device 2501 may calculate a time to scan anentire room and may accept or reject a stale map if the time to scan theentire room is above a predetermined limit based on a processingrequirement. Alternatively, the device 2501 may monitor a location ofother devices 2502, 2503 and form a request to receive a scanned imagebased on the monitored location. For example, the device 2501 maydetermine that the devices 2502, 2503 are in a different far arealocated far away and a request for a map is likely to render irrelevantdata (not useful) and formulating a request is unnecessary. In yetanother embodiment, if several devices 2501, 2502 and 2503 are presenteach device may scan different areas in the room in order to scan a roomin a more rapid manner.

FIG. 25B illustrates an embodiment method 2506 for cloud storage andretrieval of data utilized to generate a virtual object on a headmounted display. Method 2506 may be implemented in a computing devicehaving a processor configured with processor-executable instructions toperform the operation of the method 2506. Generating a virtual objectmay require intensive scanning of surfaces in a room to determine adistance of the objects, shape of the objects and the orientation ofobjects in order to anchor a virtual image to one or more surfaces inthe room. Some head mounted devices may download their location to acloud and instantly receive accurate data to assist with rendering avirtual object on a surface so as to avoid processing the data locallyand to verify scanned data. In method 2506, the processor may commenceoperation by capturing images of a user with a head mounted or bodymounted camera in block 2508. In block 2510, the processor may provideprogram instructions to scan an area of a room to capture spatial datawith a sensor array. This may be accomplished by scanning with aninfrared light emitting device and measuring a time of flight of theinfrared light reflected from the surfaces.

The data may be also provided to a processor that may generate athree-dimensional map of the room. The processor may recognize objectsin the captured image and may determine a distance parameter of theobjects in the captured image and may also determine an orientation ofsurfaces contained within the image in block 2512. The processor maygenerate a virtual object in block 2514. In an embodiment, the virtualobject 14 may resemble a collaborative flat screen three-dimensionalobject 14 that emits content. The virtual object 14 may be anchored to asurface 16 in the image and the user may move around the room but thevirtual object display parameters change so the virtual object remainsanchored to a desired anchor surface. The processor may calculateparameters including distance and orientation to display the virtualobject in block 2516 and may tag the calculated parameters and uploadthe calculated parameters to a cloud server in block 2518. The processormay tag the calculated parameters with a location tag that may be ageotag. In another embodiment, the geotag may include latitudecoordinates, longitude coordinates, altitude coordinates, bearingcoordinates, distance coordinates, and accuracy data. The Geotag addsgeographical identification metadata to various the uploaded parameters.The Geotag may help users find a wide variety of location-specificinformation in the three-dimensional map of the scanned area.

In determination block 2520 the processor may determine whether a secondhead mounted device has entered a location for which to receivecomplementary shared data to generate the virtual object faster for aparticular location. For example, a second head mounted device may be inthe same room and may emit a signal to a first head mounted device, ormay emit a signal to the cloud server, which determines that the seconddevice has entered a specific location. Processing an image can beburdensome for some devices. Also, scanning a room may also be timeconsuming. The second head mounted device, therefore, may take advantageof the work that the first head mounted device has already done andreceive the calculated parameters pertaining to objects, and surfaces inthe room to generate a virtual object for the second user faster.

If the processor determines that it has entered a new room (i.e.,determination block 2520=“Yes”), in block 2522, the second head mounteddevice may transmit its location data from a navigation device and maydownload the tagged calculated parameters from the server. In block2524, the processor of the second head mounted device may generate thevirtual object using the accessed and tagged calculated parameters. Ifthe processor determines that it has not changed locations (i.e.determination block 2520=“No”), the processor may perform operations toverify that the calculated parameters are correct in the event anotherthird head mounted device already scanned the area and generated ananchored virtual object. In block 2526, the first head mounted devicemay transmit its location from a navigation device and may downloadcalculated parameters from a server. In block 2528, the processor mayverify the calculated parameters with the downloaded calculatedparameters from the server.

In another embodiment, instead of calculated parameters includingdistance and orientation of objects, a three-dimensional map may beformed of a particular area and uploaded to the cloud server. The tag orlocation data may be embedded in the form of a header into thethree-dimensional map message and the three-dimensional map may betransmitted and stored in a cloud server for access by the second headmounted device. For example, when a second head mounted device enters anarea, the second head mounted device may upload its location coordinatesand may download a three-dimensional map of a room, or predeterminedarea. The second head mounted device may recognize objects within theimage and may determine the distance parameter of the objects within theimage utilizing the three-dimensional map. This may occur with minimalprocessing of the spatial data and the image captured at the second headmounted computing device. In another embodiment, metadata may be storedto the cloud server. The metadata may include distance and orientationof objects that were captured, which may be accessed by different headmounted devices. The metadata may also include indoor location datarelating to locations of objects in a room and an indication that theprocessor can use to determine whether the data is stale.

FIG. 26 illustrates an embodiment method 2600 for displaying a virtualobject superimposed over a detected body part, which is detected intracked images by applying and comparing an anatomical model to thetracked images. Method 2600 may be implemented in a computing devicehaving a processor configured with processor-executable instructions toperform the operations of the method 2600. In block 2601 of method 2600,the processor may commence operation by receiving data from a headmounted device including sensors that provide data regarding orientationof the device (e.g., accelerometers, gyroscopes, electronic compass,etc.). In block 2602, the processor may receive image data. In block2603, the processor may calculate the distance and orientation andprocess the content for display in block 2604 utilizing the distancesensor data, camera data, orientation data and distance data. Theprocessor may render the image with the virtual object in block 2605.

At determination block 2606, a decision may be reached regarding whetherto superimpose a virtual object over a surface of a detected body partcontained in the displayed image of the head mounted display. If theprocessor decides to superimpose the virtual object over a surface of adetected body part (i.e., determination block 2606=“Yes”), the processormay superimpose the virtual object over a body part in block 2607. Ifthe processor decides not to superimpose the virtual object over asurface of a detected body part (i.e., determination block 2606=“No”),the processor will not superimpose the object over the body part inblock 2608 and may allow the body part to occlude the virtual object ormay render the virtual object in a different location of the display.For example, the virtual object may be superimposed according to aninput by the user received by the processor.

FIG. 27 illustrates an embodiment method 2700 where a user's hands maybe occluded over a virtual object by tracking images and applying ananatomical model to the tracked images to detect body parts contained inthe image. In block 2701 of method 2700, the processor may commenceoperation by receiving data from a head mounted device including sensorsthat provide data regarding orientation. In block 2702, the processormay track images. In block 2703, the processor may apply an anatomicalmodel to the tracked image. In block 2704, the processor may identifythe user's hands and render the image in block 2706. For example, theprocessor may identify fingers, a hand, and an arm that originate from abottom upwardly in the tracked images as a user's hand. While at thesame time, an anatomical or skeletal tracking algorithm may be appliedto detect a second body part. The second body part may also includefingers, a hand, and an arm. The processor may infer that a second bodypart originating at a top and extending downwardly in the image may beanother user's body part. Additionally, the processor may infer thatarms and hands that are a predetermined distance too far away from theuser based on size and orientation in the image may be another user'sdetermined body part.

At determination block 2706, a decision may be reached regarding whetherto occlude the user's hands over the virtual object. If the processordetermines to render the user's hands over the virtual object (i.e.,determination block 2706=“Yes”), in block 2707, the processor may renderthe virtual object with selective occlusion of the user's hands over thevirtual object. For example, the user's hands may block the virtualobject from the viewer. In block 2708, the image may be updated by theprocessor. If the processor determines not to render the user's handsover the virtual object (i.e., determination block 2706=“No”), in block2709, the processor will display the virtual object superimposed overthe user's body part so the body part appears transparent. For example,the user may run his hand over the virtual object. However, the virtualobject may be displayed over the body part and superimposed over thesurface of the hand so the content on the virtual object remains seen,despite the fact that the user's hand should be blocking the virtualobject. Once a command is detected and confirmed from the user by adetected gesture, or input command, the input results in execution ofthe task and action of the detected command and the virtual objectcontent may be updated in block 2709.

FIG. 28A illustrates an embodiment method 2800 where a secondindividual's hands may be rendered transparent or absent over a virtualobject. Method 2800 may be implemented in a computing device having aprocessor configured with processor-executable instructions to performthe operations of the method 2800. In method 2800, the processor maycommence operation by receiving data from a head mounted deviceincluding sensors that provide data regarding orientation in block 2801.In block 2802, the processor may track captured images. In block 2803,the processor may apply an anatomical model to the tracked image. Forexample, the anatomical model may be a skeletal tracking algorithm thatcompares tracked objects to a skeletal model to determine if there areany body parts contained in the image. If an object is not detected tobe a body part by the processor, the processor may ignore the object.

In block 2804, the processor may identify the user's hands and a secondindividual's hands. The processor may render the image in block 2805 onan anchor surface contained in the image, for example, on a desktop oron a wall. At determination block 2806, the processor may determinewhether to superimpose the virtual object over a body part. If theprocessor determines to superimpose the virtual object over a body part(i.e., determination block 2806=“Yes”), in block 2807, the processor mayrender the second individual's hands transparent or absent over thevirtual object, or when the body part occludes the virtual object, thevirtual object may be displayed over a top surface of the body part sothe body part appears to be transparent. In another embodiment, anoutline of the body part may still remain visible with virtual objectsuperimposed over a body part surface. In block 2808, the displayedimage of the virtual object may be updated by the processor to accountfor changes in the content and movement of the body part. If theprocessor determines not to superimpose the virtual object over a bodypart (i.e., determination block 2806=“No”), the processor will displaythe virtual object in block 2809 and update the virtual object formovement of the user.

FIG. 28B illustrates an embodiment method 2810 where a processor mayinfer which hands are the user's hands and which hands are other user'shands and the processor may render the other user's hands as transparentor absent when the hands occlude a virtual object. Method 2810 may beimplemented in a computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 2810. In method 2810, the processor may commence operation byreceiving data from a head mounted device including captured image datain block 2811. In block 2812, the processor may capture spatial datawith a sensor array. In block 2813, the processor may recognize objectsin the image and determine a distance parameter of the objects in theimage. In block 2814, the processor may process images to track objectsand the processor may apply an anatomical analysis via a skeletal modelto the tracked image. This may include identification of a partialanatomy when the camera is mounted on the user or where another user isonly partially visible in the field of view of the sensor. For example,the anatomical model may be a skeletal tracking algorithm that comparestracked objects to a skeletal model to determine if there are any bodyparts contained in the image. If an object is not detected to be a bodypart by the processor, the processor may ignore the object. For example,the processor may apply an anatomical analysis to the image to determinewhether a body part is within the image. The processor may reference astored anatomical or skeletal model within a memory and compare it tothe tracked objects in the video image. The stored skeletal model mayinclude anatomical data, for example, a relative position of key joints,bones and limbs in a human skeleton, and other data. The processor maycompare the tracked objects in the image to the stored skeletal model todetermine if the body part is in the image.

For example, the processor may detect the user's arm by detecting an armand a number of fingers extending from a hand in a first orientation, orwith an arm, a wrist and fingers progressing from a bottom of an imageupwardly. The processor may detect the second individual's arm bydetecting a second arm and a second number of fingers extending from thesecond hand in a second different orientation, or with an arm, a wristand fingers progressing from a top of the image downwardly. In anotherembodiment, the processor may distinguish from between the user andanother's body parts by reviewing the tracked images for orientation,object size, distance from the user, or by counting the number of bodyparts within the image.

In block 2815, the processor may render the image with an anchoredvirtual object in the image. At determination block 2816, the processormay determine whether the detected body part belongs to the user or toanother individual. The processor may superimpose a virtual object overanother user's body part. In the displayed image only the user's handsocclude the virtual object while other hands, arms and body parts willnot occlude the virtual object and will appear transparent.

The user may see the virtual object as superimposed over the otheruser's body part so the other user's body part is rendered astransparent. The processor may determine an area in the image relevantto the other individual's body part and may display the virtual objectover the image only when the other individual's body part occludes theanchored virtual object. If the processor determines that the detectedbody part belongs to the user (i.e., determination block 2816=“Yes”), inblock 2818, the processor may render the user's body part as solid whenthe user's body part occludes the image in block 2817. If the processordetermines that the detected body part belongs to another individual,(i.e., determination block 2816=“No”), the processor will display thevirtual object, or a portion thereof as superimposed over the detectedbody part in block 2819 when the body part occludes the virtual object.

For example, the processor may determine an area in the image defined asan area (in pixels) over the body part of the other individual. Theprocessor may form the superimposed virtual object over the area and notrender the virtual object over the perimeter of a body part. When thebody part is located over the anchored virtual object an outline of thebody part of the second individual may still be shown and rendered.Therefore, the outline may be visible over the anchored virtual objectwhile a remainder may be rendered as transparent. In block 2819, theprocessor may update the virtual object for movement of the user toaccount for changes in the content and movement of the body part. Forexample, the processor may display the anchored virtual object as arectangular virtual display on a surface. As described above, theprocessor may analyze captured images to detect user gestures. Aseemingly virtual reality interaction with the anchored virtual objectby the second individual and the user may occur. In another embodiment,the processor may detect a body part of a third individual. Theprocessor may display a superimposed virtual object over the body partof the third individual when the body part of the third individual islocated over the anchored virtual object. The processor may also notdisplay the superimposed virtual object over the body part of the thirdindividual and allow the body part to be displayed when the body part isdetermined to be located off the anchored virtual object and notoccluding the virtual object.

FIG. 29 illustrates an embodiment method 2900 where data may betransmitted from a server to multiple devices to provide services and toalleviate processing at the head mounted device. The method 2900 may beimplemented within a system like that described above with reference toFIG. 25A, which may include a server and one or more head mounteddevices configured with processor-executable instructions to perform theoperations of the method 2900. In method 2900, the server 2504 mayreceive image and location data from a head mounted device includingsensors that provide data regarding spectral data and images in block2901. In block 2902, the processor may upload data to the server withposition information relating to a location where the data was scanned.The server may output scanned data to a number of different head mounteddevices. In block 2903, the processor may output data to a second headmounted display as a reference.

In block 2904, the processor may output data to a third head mounteddevice to supplement the third head mounted device's scanned data and toverify the captured scanned data. In block 2905, the server may outputdata to a fourth head mounted device to enhance the data present on thedevice and in block 2906, the server may upload data to conserve batterypower and processing. Substantially simultaneously, each of the headmounted devices may be conducting their own scanning of the area todetermine information about objects in the area. For example, the headmounted devices may scan the area to determine a shape and dimensions ofobjects in the image, and may scan the area to determine an orientationof the objects in the image and relative positions of surfaces in theimage, which may be useful to anchor a virtual object for an immersivevirtual or augmented reality experience. At determination block 2907,the processor may determine whether to upload the updated scannedpositional data from the devices to the server for storage on theserver. If the processor determines to upload scanned positional data tothe server (i.e., determination block 2907=“Yes”), the processor mayupload updated data and merge and combine the head mounted devicesscanned data with the data stored on the server. If the processordetermines not to upload scanned positional data to the server (i.e.,determination block 2907=“No”), the processor may display the virtualobject in block 2908 and update the virtual object for movement of theuser.

In another alternative embodiment, the processor of FIG. 29 may operateas a ‘magic lens’. In this manner, the virtual object displayed on aphysical surface in a room may be viewed on a conventional smartphone orcomputing device. Such a smartphone may output data to a server and mayfurther link to the server. A user, without a head mounted device, andusing a mobile communication device, may collaborate using the mobilecommunication device or smartphone with users wearing the head mounteddisplays. For example, a user may view an output of a virtual objectanchored on a physical surface by watching a smartphone display and maycommunicate via a wireless audio link to the user utilizing a headmounted display.

The smartphone may be operatively connected to the server via a wirelessRF link, such as a cellular data link (e.g., 3G, 4G, LTE, UTMS, etc.).The smartphone may output an image on a display of the smartphone. Theimage may correspond exactly to another user's head mounted displayoutput. The user viewing the display on the smartphone may view theaugmentations and virtual objects shown on physical surfaces. Thisdisplay may occur in two or three dimensions on the display of thesmartphone. The smartphone may also be operable to collaborate with thehead mounted device 10 using a mobile communication device and awireless interface to receive signals from the server. Alternatively,the smartphone may receive RF data signals using a short rangecommunication protocol directly from the head mounted device 10. In thismanner, the user may hold up the smartphone and view the virtual oraugmented reality experience of the users of the head mounted displayfor collaboration. In this embodiment the smartphone may manipulate theimage to correct for the smartphone's own position and orientation inrelation to near by surfaces using methods similar to those describedherein.

FIG. 30 illustrates an embodiment method 3000 where users may becollaborating using two head mounted devices. The two head mounteddevices may be running different incompatible software applications fromtwo different software providers. Generally, a first user using a firstword processor cannot work with another user using a second differentincompatible word processor at the same time in a collaborative fashion.In method 3000 a processor may reconcile and translate data so a firstuser using a first software application and a second user using a secondsoftware application can collaborate with one another and enter datathat is viewable in real time.

The method 3000 is not limited to a head mounted device and may beimplemented in other types of computing device. In method 3000, theprocessor may commence operation by opening a first application in block3001 and opening a second application in block 3002. At determinationblock 3003, the processor may decide whether to collaborate with otherdevices. If the processor decides to collaborate with other devices(i.e., determination block 3003=“Yes”), in block 3004, the processor maycollaborate and provide a two way communication between the devices. Ifthe processor decides not to collaborate with other devices (i.e.,determination block 3003=“No”), the processor will display the virtualobject in block 3010 and update the virtual object for movement of theuser.

In block 3005, the processor will render a virtual screen as the virtualobject. The virtual screen may provide a display output of a softwareapplication. The virtual screen updates for changes entered by at leasttwo different users. In block 3006, the processor will apply a conflictresolution program for two different software applications. The conflictresolution may provide a translation of the inputs and outputs of thetwo incompatible software programs to a generic format, which may beloaded to a third generic software program, which reconciles changes inthe generic format, and translated the reconciled data back into thefirst and second incompatible formats. The third software program mayprovide a translation of the generic format back to the first and thesecond software programs to render the changes, and which can beviewable by the users in the original native program.

For example, data may be collected from a first software programpertaining to the inputs of the first user and communicated to a genericformat and communicated to a third software program. Next, the data maybe collected from a second software program pertaining to the inputs ofthe second user and communicated to a generic format and communicated tothe third software program.

The third software program may reconcile the changes and may store thechanges in a memory. Now, each of the users would like to view eachother's changes to a document in real time. The third software programmay translate the changes into a format understood by each of the firstand the second software programs. The third software program may outputthe translated changes back to the first and the second software programso each of the users may view the total changes in their nativeprogram's formatting. In block 3007, the processor may also identify anauthor of the changes. In block 3008, the processor will update thedocument map and render the document image as the virtual object inblock 3009 and the processor may update the virtual screen in block3005.

FIG. 31 shows a high level block diagram of three different usersoperating a computing device shown as reference numeral 3100, 3101 and3102. Each may be using a different software application. Data from eachof the software programs may be stored on the server 3103. For example,first local changes may be made in the relevant software program asconventionally known. Second remote changes may also be made and writtento a shared storage medium 3106. For example, the second remote datachanges may be communicated to a library 3104. Library 3104 may convertthe input data and output data from each application from the remotechanges into a generic format that can be input and read by a genericsoftware program. The generic format may be stored on a library 3104 andmay be written to a shared document format on storage medium 3106.

The shared document format stored on the storage medium 3106 mayaggregate the changes into a single generic program. The single genericprogram may also translate and output the changes rendered by each userin a single stream of data back to each of the software application in aformat that may be specific to and understood by the relevant softwareapplication so each user 3100, 3101, and 3102 can view changes made byother users.

For example, the shared document format 3106 may include a conflictresolution functionality that may provide a translation of the inputsand outputs of the software programs to a generic format, which may beloaded to the third independent software program 3106. There, thechanges from all of the users are entered and stored. A translation of anumber of different software word processing programs may be stored onthe server 3103. The processor may provide a translation of the genericformat having all of the changes back to the first, the second and thethird software programs 3100, 3101 and 3102 to render and illustrate thechanges therein.

FIG. 32 illustrates an embodiment method 3200 where users may becollaborating using two computer systems. The users may be collaboratingwith one another utilizing different incompatible software applicationsfrom two different software providers. The method 3200 may reconcile theinputs and outputs of the data so any and all changes are viewable bymultiple users utilizing multiple different incompatible computersoftware applications.

In method 3200, the processor may commence operation by opening a firstapplication in block 3201 and opening a second application in block3202. At determination block 3203, a decision may be reached regardingwhether to collaborate from the devices to a server. If the processordetermines to collaborate with a server (i.e., determination block3203=“Yes”), the processor outputs program instructions for acollaboration and provide a two way communication between the devices(block 3204). For example, changes on each of the incompatible softwareapplications may be rendered locally and also transmitted to a remotelocation, where the changes may be aggregated and entered into a thirdsoftware program. If the processor determines not to collaborate with aserver (i.e., determination block 3203=“No”), the processor will notestablish the two way communication between the devices and render thescreen in block 3210 and update changes made in a non-collaborativemanner.

In block 3205, the processor will render a screen and update the screenfor changes. For example, changes on each of the software applicationsmay be transmitted to the remote location, where the changes areaggregated and entered into the third compatible software program. Inblock 3206, the processor will apply a conflict resolution program fortwo different software applications. The changes are then translatedinto a native format, and then are transmitted to each of the users inthe native format for display on the respective applications. In block3207, the processor will identify an author of the changes. In block3208, the processor will update the document image and render thedocument image. In block 3209, the processor will update the documentimage.

FIG. 33 illustrates an embodiment method 3300 where an audio command maybe processed and confirmed using a detected gesture. Method 3300 may beimplemented in a computing device having a processor configured withprocessor-executable instructions to perform the operations of themethod 3300. In method 3300, the processor may commence operation byreceiving scanned data from sensors that provide data regardingorientation of the head mounted device in block 3301. In block 3302, theprocessor may receive image data from the camera regarding images. Inblock 3303, the processor may calculate distance and orientation ofobjects within the images. In block 3304, the processor may process theimage using data from sensors, camera data and distance data and inblock 3305 the processor may render the image with the virtual object.Additionally, in block 3305, the processor may track the user'smovements to determine if there is a gestural input command that is tobe executed. In block 3306, the processor may receive an audio signal.

At determination block 3307, the processor may determine whether anaudio command spoken by the user is recognized. For example, the usermay desire to enlarge a size of the virtual object to a larger virtualobject rendered on the display and may speak the word “enlarge virtualobject.” For example, the user may alternatively wish to change a shapeof the virtual object from rectangular to a different shape and mayspeak a word “change virtual object to oval”. Other similar commands arealso envisioned and possible and within the scope of the presentdisclosure. If the processor recognizes an audio command spoken by theuser (i.e., determination block 3307=“Yes”), in block 3308, theprocessor may track video data for a gesture for a predetermined timeperiod. The gestural command may be utilized to confirm the first audiocommand. For example, the user may be speaking and may simplyinadvertently mention “enlarge” and “virtual object” in a sentencespeaking to a second user, which may be received by the microphone andcommunicated to a processor. Thereafter, instead of blindly executingthe command, a confirmation may be awaited in the form of a detectedgesture. For example, the user may point in a direction or the user maygive an “O.K.” sign with a thumb making an index finger and theremaining fingers being outstretched. This second command will confirmthe first command. At determination block 3309, the processor maydetermine whether a gesture is recognized confirming the audio command.If the processor recognizes that a gesture is confirming the audiocommand (i.e., determination block 3309=“Yes”), the processor mayexecute the task in block 3310.

If the processor does not recognize an audio command spoken by the user(i.e., determination block 3307=“No”), the processor may continue toreceive audio. If the processor does not recognize that a gesture isconfirming the audio command (i.e., determination block 3309=“No”) theprocessor may continue to receive and analyze audio data. For example,if the user may be speaking and may simply inadvertently mention“enlarge” and “virtual object” in a sentence speaking to a second user.However, if no confirmation is detected in the form of the detectedgesture (pointing, O.K. sign, outstretched hand), the processor will notexecute the first command and simply may continue to monitor audio foradditional new commands Many gestures may provide a confirmation and arenot limited to any disclosed herein. Also, verbal commands (e.g., viaspeech recognition) and commands entered by user button presses (e.g.,on a keyboard) may be accepted and processed by the processor. Forexample, verbal commands may be accepted to confirm or express the sameinformation as gesture commands, thereby enabling the head mounteddevice and system to more accurately interpret a user's intent.

FIG. 34 illustrates an embodiment method 3400 where a gesture commandmay be processed and confirmed using a detected audio command, which maybe a different order of operation than disclosed above. Method 3400 maybe implemented in a computing device having a processor configured withprocessor-executable instructions to perform operations of the method3400. In method 3400, the processor may commence operation by receivingdata from a head mounted device including sensors that provide dataregarding orientation in block 3401. In block 3402, the processor mayreceive data from the camera regarding images.

In block 3403, the processor may calculate distance and orientation ofobjects within the images. In block 3404, the processor may process animage using data from orientation sensors, camera data, and distancedata. In block 3405, the processor may render the image with the virtualobject and in block 3406 the processor may control the camera to trackthe user's movements.

At determination block 3407, a decision may be reached regarding whetherto recognize a gesture command, for example, by recognizing one or moreselected gestures including pointing, clapping, snapping fingers, makingan “O.K. gesture” by joining the thumb and finger. This may be performedusing an anatomical model that may be applied to the tracked video data.If a gesture is detected (i.e., determination block 3407=“Yes”), theprocessor may track audio data for a command for a time period in block3408 to confirm the detected gestural command.

At determination block 3409, a decision may be reached regarding whetherto recognize an audio command as a confirmation of the gestural command.If the confirmation is received (i.e., determination block 3409=“Yes”),the processor may execute the task in block 3410.

If a gesture is not recognized (i.e., determination block 3407=“No”),which may occur when the gesture is inadvertently detected, theprocessor may continue to track the captured video images for a newgestural command. Also, if no audio confirmation is recognized (i.e.,determination block 3409=“No”), the processor may continue to receivevideo and track the video and ignores the gestural command as noconfirmation was detected. Also, mentioned above verbal commands andcommands entered by user button presses may be accepted and processed bythe processor, alone or in conjunction with gesture commands in order toenable the head mounted device and system to more accurately interpret auser's intent.

FIG. 35 illustrates an embodiment method 3500 where a gestural commandmay be processed and recognized. In response to the gestural command,the processor may render a “virtual input device” on the virtual objectso the user can utilize the virtual input device to manipulate thevirtual object. The virtual input device may be a generated virtualobject that can be controlled by tracking the movements of the user. Forexample, the virtual object may be a virtual pointer, or an icon or anyother controllable virtual object selected by the user.

Method 3500 may be implemented in a computing device having a processorconfigured with processor-executable instructions to perform theoperations of the method 3500. For example, the input device may be avirtual reality pointer object. The pointer may be generatedautomatically in a line of sight measured from the gesture/trackingcameras and a virtual object connected to the anchor surface. In method3500, the processor may commence operation by receiving data from a headmounted device including sensors that provide data regarding orientationin block 3501. In block 3502, the processor may receive captured imagedata from the camera regarding stereo images.

In block 3503, the processor may calculate distance and orientation ofobjects within the images. In block 3504, the processor may processimage using data from orientation sensors, camera data and the distancedata. In block 3505, the processor may render the image with the virtualobject and in block 3506 the processor may control the camera to trackthe user's movements.

At determination block 3507, the processor may determine whether agesture command is recognized, such as by applying an anatomicalrecognition algorithm to the tracked images. The processor may recognizeone or more selected gestures including pointing, clapping, snappingfingers, making an “O.K. gesture” by joining the thumb and finger, etc.within the image. If the processor recognizes a gesture (i.e.,determination block 3507=“Yes”), the processor may have an indication toexecute a virtual pointer task in block 3508.

The pointer may be an “input device related virtual object” (a mouseicon, a laser pointer virtual object, an arrow input device) that isdisplayed on the display in three dimensions and that is placed on thevirtual object and that can allow a user to highlight text, select, makeinput commands and provide inputs to the processor of the head mounteddevice. At determination block 3508, a decision may be reached regardingwhether to execute the pointer task. If the processor determines toexecute the pointer task (i.e., determination block 3508=“Yes”), theprocessor may execute the task and a type of pointer may be selectedfrom a plurality of different virtual input devices in block 3509. Onblock 3510, a vector may be calculated from the head mounted or bodymounted camera to the user's finger and to the virtual object. Thevector may be used to render the pointer on the virtual object in block3511. In block 3512, the virtual object and the virtual input device'sdisplay may be updated.

If the processor does not recognize a gesture (i.e., determination block3507=“No”), the processor may continue to process video images in orderto recognize predetermined gestures. Also, if the processor determinesnot to execute the pointer task (i.e., determination block 3508=“No”),the processor may continue to track video and ignore the gesturalcommand as no task will be executed.

FIG. 36 illustrates an embodiment method 3600 where a gesture commandmay be recognized and processed and a three-dimensional virtual realitytype input device may be rendered on the virtual object to provide inputcommands. Additionally, the processor may recognize features ofindividuals, which may be rendered transparent when the features/bodyparts occlude the virtual object. Method 3600 may be implemented in acomputing device having a processor configured with processor-executableinstructions to perform the operations of the method 3600. For example,the input device may be a virtual reality pointer object that is onlydisplayed on the head mounted display. The object may be displayed onthe display and may be generated in a line of sight or in a vector thatmay be measured from the gesture/tracking cameras to the virtual objectthat is connected on the anchor surface. For example, a vector may beformed from the camera/or a user's finger to the virtual object on thedisplay. The virtual pointer object may be placed and displayed asconnected to the vector. The user may control the virtual pointer objectby one or more detected gestures, audio commands, by an input device, orby another input command. In method 3600, the processor in block 3601may commence operation by receiving data from the camera regardingstereo images.

In block 3602, the processor may calculate the distance and theorientation of the objects within the images. In block 3603, theprocessor may process the image using data from orientation sensors,camera data and distance data. In block 3604, the processor may renderthe image with the virtual object. In block 3605, the processor mayprovide instructions to control the camera to track the user's movementsby applying an anatomical algorithm to the tracked images to a detectedbody part and may track the body part over a predetermined timeinterval.

At determination block 3606, the processor may decide whether gesturecommand is recognized, which may occur when there is a match among oneor more gestures in a gesture dictionary with shapes and movement withinthe tracked images, the processor may correlate the detected gesture toan input command to execute a software application, or to take somepredefined action. If the processor recognizes a gesture command (i.e.,determination block 3606=“Yes”), the processor may execute a pointertask in block 3607.

The pointer may be a virtual input device (a mouse icon, a laser pointervirtual object) that is displayed on the head mounted display. Thevirtual input device may be associated with the virtual object that canallow a user to highlight text, select hyperlinks, formulate inputcommands and form inputs relative to the virtual object. For example,the virtual input device may be a three-dimensional virtual realityarrow displayed on the head mounted display. In such an embodiment, thearrow may be moveable in free space on the display in response to usergestures. For example, in response to the user moving a hand to theleft, the system may render an image of the arrow moving to the left onthe head mounted display. As another example, the user may input acommand and the arrow may illustrate that the command is executed. Forexample, the user may speak the word “select” when the arrow is adjacenttext on the virtual object and the arrow may indicate that the text isselected.

At determination block 3607, the processor may determine whether toexecute the pointer task and whether to select a virtual input deviceobject from a number of predefined alternatives (e.g., an arrow, apointer device, or various virtual icons). If so (i.e., determinationblock 3607=“Yes”), the processor may execute the task and a virtualinput device may be selected in block 3608 from an array of differentpointer types stored in memory. In block 3609, a vector may becalculated from the camera to the user's hand and to the virtual object.The virtual input device may be rendered and placed directly on thecalculated vector and may be moveable on the display by trackingmovement of the user's hand to provide for a virtual or augmentedreality experience. In block 3610, the virtual reality inputdevice/pointer may be used to move, select, enlarge or highlight aportion of the virtual object on the display.

In block 3611, optionally a user's features may come in contact with thevirtual object and may occlude the virtual object. For example, a user'shand may pass over the virtual object and may occlude or block the viewof the virtual object from the user's view point, which may not befavored. The processor may superimpose the virtual object on the user'sbody part so the virtual object may remain visible to the user, eventhough the user's body part should be occluding the virtual object onthe display. In this manner, the body part may be rendered as seeminglytransparent with only an outline of the body part shown. In block 3612,the vector may be used to render the pointer on the virtual object. Inblock 3613, the virtual input device may be updated. In block 3614, thevirtual input device may be displayed with the virtual object.

If no gesture command is recognized (i.e., determination block3606=“No”), the processor may continue to track video to determine ifthe processor recognizes a gesture. Also, if the processor decides notto execute or use a pointer (i.e., determination block 3607=“No”), theprocessor may continue to track video and will not generate and displaya virtual input device.

FIG. 37 illustrates a high level illustration of a system 3700 thatincludes a virtual object reality input device or a virtual pointer 3706that is visible on a head mounted display 3704 and a high level diagramof the virtual object 14. The head mounted display 3704 may include aprocessor and a wireless communication device that includes an antenna(not shown). As discussed above with reference to FIG. 5A, the headmounted device 3704 may include a core processor which may include acoprocessor for executing applications, sometimes referred to as anapplication processor.

The head mounted device 3704 may also include multiple peer processorsassociated with important device capabilities including communications(e.g., a wireless modem processor), navigation (e.g., a GPS receiverprocessor), and graphics (e.g., a graphics processor unit (GPU)). Thehead mounted device 3704 may also include a cellular transceiver ormodem (not shown), which in many cases include one or more processorswhose main task is managing wireless communications. The head mounteddevice 3704 may also include other wireless transceivers forcommunicating over other types of wireless communications networks, suchas a Wi-Fi modem (not shown) that may be a separate transceiver chip, inwhich case it may also include a separate programmable processor. Otherwireless transceivers (not shown) which have embedded programmableprocessors may be included to connect to other types of wirelesscommunication links, such as near field communications (NFC) links,Bluetooth® links, and links based on other wireless protocols.

In addition to the communications processors, the head mounted device3704 may also include a GPS receiver (not shown) which may have adedicated processor. Also, a graphics processor may be included in orderto provide high-resolution graphics. The head mounted device 3704 mayalso include processors associated with wired communications, such as aUSB port. As described above with reference to FIG. 5A, the head mounteddisplay 3701 may include an RGB-D camera system which itself may combinea RGB camera 3704 a with an infrared camera 3704 b and a structuredinfrared light infrared emitter 3704 c.

The processor measures a vector that may be captured from the headmounted or body mounted camera 3701 to the user's finger 3705 and fromthe finger 3705 to the virtual object 14. Using the vector an inputdevice virtual object 3706 may be generated and may be placed on thevector. The virtual input device 3706 may be generated and anorientation of the virtual input device 3706 may be calculated. Thevirtual input device 3706 may be generated to connect to the anchorsurface 3703, which may be where a virtual object 14 is connected. Inthis embodiment, the virtual object 14 may be rendered for display to avery large size, for example the size of a building as displayed on thehead mounted display. The user using the virtual input device 3706 mayselect text or portions of the virtual object or content displayed withthe virtual object 14 thereof.

FIG. 38 illustrates a high level diagram of a first and a second headmounted device system 3800 and 3800 a illustrating how a second headmounted device may provide pointing coordinates and gesture controls forimplementation by a first head mounted device. The primary head mounteddevice system 3800 may include a first head mounted or body mountedcamera array 3801 and a first processor 3804 operable with a peer topeer based arrangement. A first processor 3804 may be operable as a hostprocessor and a second processor 3805 may be operable as a secondarysystem. A second head mounted device system 3800 a may act as systemthat receives a generated image for display from a primary system 3800.The second head mounted device system 3800 a may include a head mountedor body mounted camera array 3802 and a second processor 3805. A displaysystem 3803 may be also be provided for the user to viewthree-dimensional images. The head mounted or body mounted camera array3801 may provide the image and distance data to the first processor3804, which generates and provides the image data to the display system3803. The primary processor 3804 may provide the image to the secondprocessor 3805, and the second head mounted device may provide pointcoordinates and gesture controls to the primary processor 3804. Imageand distance data and provided to the processors 3804 and 3805 whereby aprimary processor generates a virtual object and displays the image on adisplay system 3803 and also communicates the image to the secondprocessor 3805 for display. The second processor 3805 may detect usergestures as user inputs, such as pointing to locations for anchorsurfaces or moving objects and control commands, and provide those pointand gesture controls to the first processor where the inputs may beprocessed.

FIG. 39 illustrates a system 3900 that operates in a cloud configurationwhere point and gesture controls and image data information may bestored on and relayed to others (e.g., in a multicast) by a cloud server3906. This system enables virtual object and display data to bewirelessly communicated to collaborating head mounted display systems3900 a and 3900 b. The system 3900 also enables other computing devices,including other head mounted devices, to monitor the data exchangestaking place between the collaborating head mounted display systems 3900a and 3900 b. In this manner, the collaboration may be viewed by a wideraudience.

The system 3900 may include a central processor operable on a centralserver 3906. The first head mounted device 3900 a may include a headmounted or body mounted camera 3902 and a first processor 3903. A secondhead mounted device 3900 b may include a head mounted or body mountedcamera 3904 with a second processor 3905. A display system 3901 also maybe provided. An image may be provided from the server 3906 to thedisplay system 3901. The image also may be provided to the firstprocessor 3903. The first processor 3903 transmits data to the server3906 including gestures, point coordinates, distance data, image data,orientation data, and sensed parameters. The image may be also providedto the second processor 3905. The second processor 3905 also may providethe data including gestures, point coordinates, distance data, imagedata, orientation data, and sensed parameters to the server 3906. Thecentral processor operable on the central server 3906 may provideservices to the head mounted devices to alleviate computationalprocessing on the first and second processors 3903 and 3905 to renderthe virtual pointer object and a virtual object.

For example, the first user system 3900 a may scan an area with infraredlight and may measure a time of flight of the reflected infrared lightto a gating mechanism. This may be utilized to determine the shape ofobjects in a predetermined area and may also be utilized to determine adistance parameter of the objects. The first user system 3900 a maywirelessly communicate the distance parameter to the sever 3906. Thecloud server 3906 may utilize the scanned data to generate an imageincluding a virtual object. The virtual object may be generated in animage and wirelessly communicated to the display system 3901 and thesecond head mounted device 3900 b together with scanned data of thepredetermined area.

Referring back to FIG. 37, an example of a gesture that may be trackedby a head mounted device is a pointing gesture indicating or selecting avirtual object. For example, a user may point to a virtual object 14with a finger 3705, and the camera system of the head mounted displaymay determine a vector from the head mounted or body mounted camera 3708to the user's finger 3705 and from the finger 3705 to the virtual object14. Using this vector, the processor may generate an input devicevirtual object.

For example, an orientation of the virtual object 14 for display on thehead mounted display may be determined. The virtual input device object14 may be connected to the anchor surface 3704 and may connect to thevirtual object 14 so the user may control the virtual input deviceobject 14. The virtual object 14 may be anchored on the anchor surface3703 within images rendered on the head mounted display. In thisembodiment, the virtual object 14 may be rendered for display, and usingthe virtual object input device, the user may select text or portions ofthe virtual object 14 which are recognized with the tracked video andcaptured gestures utilizing the gesture tracking camera 3708. In thisembodiment, the virtual input device object 14 may be displayed as amoveable three-dimensional object around the user or that the user canwalk around. The virtual object 14 may be rendered as a very largevirtual object in the displayed image, such as the size of a building.The user may utilize the virtual input device to specify a position inspace on the very large virtual object 14 in the displayed image.

FIGS. 40A-40B illustrate another embodiment in which the head mounteddisplay device 10 shown in FIG. 1 may be replaced or supplemented with apico projector 4005 that may be wirelessly connected to a mobilesmartphone 4010 and that may generate a projected image on a surfacewhere a user's gestures may be utilized to provide input commands. Inthis embodiment, the user does not wear any head mounted apparatus andmerely may view the projected images on a horizontal or verticalsurface. A handheld projector 4005 (also known as a pocket projector4005, mobile projector or pico projector) may be used with a handhelddevice 4010 to output the virtual object 14 a shown in FIGS. 40A and40B. The handheld projector 4005 may include miniaturized hardware andsoftware that can project digital images 14 a onto any nearby viewingsurface 16, such as a wall 16 shown in FIG. 1 or a desk top 16 a shownin FIG. 40B.

The pico projector 4005 may include a battery 4015, an electronicmotherboard 4005, a diode laser, laser or other light source 4030, acombiner optical element 4035, and scanning mirrors 4040. Variousconfigurations of components in the handheld projector 4005 are possibleand within the scope of the present disclosure. The processor 4020 maybe coupled to a motherboard 4020. Processor 4020 may control the laserdiode 4030 and a projector to output an image where controlled by theprocessor 4020 utilizing the mirror and optic 4040 and 4035.

Turning to FIG. 40A, the pico projector 4005 may include a projectorelement 4006, an input sensor 4007, a connector 4008, and a wirelessinterface 4009 for connecting to a mobile device 4010 or other computingdevice. An input-sensing pico-projector may be connected to other unitsof the same type in a modular fashion. The projectors may createmultiple, positional aware shared projected displays rendered as virtualobjects 14 a and 14 b for a real time local digital collaborationbetween users. The pico projector 4005 may capture images for a firstuser and applying an anatomical algorithm to the captured images. Theprocessor may detect a first user and project a first virtual object 14a on a desktop surface 16 in a convenient location for the first user.The pico projector 4005 using a peer-to-peer or other RF wirelesscommunication link, or using captured image data may detect that asecond user has joined the collaboration. The pico projector 4005 mayoutput a second virtual object 14 b oriented correctly for the seconduser based on the captured image data.

Each projector 4005 may connect to a mobile computing device 4010 (orother computing device) via high-speed wireless data links (e.g., 4G,Wi-Fi or Bluetooth®) and/or be linked via a peer-to-peer communicationprotocol. Each projector 4005 may also be linked to the mobile computingdevice 4010 (or other computing device) via a wired data link. Themobile computing device 4010 may act as a processing unit and datastorage device for the pico projector 4005 and may generate theprojected image and communicate the image to the pico projector 4005.The pico projector 4005 may alternatively connect physically to otherpico projectors, such as via strategically positioned connectors 4008 oneach projector's 4005 housing. For example, two pico projectors 4005 mayconnect via respective connectors 4008. The connectors 4008 may providea reference data point for each of the pico projectors 4005 when twopico projectors 4005 are connected to one another. The pico projector4005 may display a projected image on a surface 16 a, which may be apredetermined distance from the reference data point.

The connections 4008 on each projector's 4005 housing may provide bothphysical support and may include a circuit for a high-speed data linkbetween projectors 4005, bridging communications between eachprojector's wirelessly connected mobile computing device 4010. Theconnector 4008 and circuit may support USB, Firewire®, or any other highspeed connector 4008 known in the art.

Additionally, the connections may provide a reference data point tosoftware regarding each projector's physical orientation with respect toother projectors connected via connector 4008 shown in FIG. 40A. Theconnectors 4008 may be used to provide interface software and theability to transfer digital assets across the projectors 4005. Theprojector 4005 may output the virtual object 14 a as a projection onto asurface. The projector 4005 may also use captured image data to senseobjects in the image for tracking and for gesture recognition. Theprojected image 14 a may be a virtual touch screen 14 a and may alsoinclude a virtual input device on the virtual object. Gesturesassociated with the projected image may be detected and the processormay formulate input commands based on the detected gestures. Thisgesture recognition functionality may provide a simulated appearancethat the user may be controlling a virtual input device.

FIG. 40B illustrates the pico projector 4005 rendering a first virtualobject 14 a and a second virtual object 14 b. In an embodiment, a singlepico projector 4005 may project at least two different images for eachuser. In another embodiment, the pico projector 4005 may be mated with asecond pico projected which completes an electronic connection betweenthe two projectors. The two different pico projectors 4005 maywirelessly connect with a mobile computing device 4010 or otheravailable computing devices, and each may output two different imageswith one for each user. The projector 4005 linked to the mobilecomputing devices 4010 may use the processor of each mobile computingdevice 4010 to render the virtual object 14 a and 14 b. The processor ofthe mobile computing device 4010 and the image sensor may detectparameters associated with each user to render the virtual object 14 aand 14 b and to determine the correct orientation of the virtual object14 a and 14 b for each of the users.

The input sensor 4007 may convert the image into an electronic signal.The electronic signal may drive the light sources with different colorsand intensities down different optical paths. In the combiner optic 4035the different light paths may be combined into one path demonstrating anumber of colors. The mirrors 4040 reflect the image pixel-by-pixel andproject the image. The handheld projector 4005 may project a clearimage, regardless of the physical characteristics of the viewing surface16. For example, the projected image of the virtual object 14 a may be a“virtual object input device,” such as a keyboard with a word processingapplication. The gestures of the user may be detected corresponding tothe input device locations on the virtual objects 14 a-14 c andrecognized as one or more input commands.

In the illustrated embodiment, the image may project a virtual object 14a on a surface 16 a shown in FIG. 40B, and the user may anchor thevirtual object 14 a on the physical surface 16 a. As the user movesbased on the output of the sensor 4007, the virtual object 14 a maychange in shape, orientation and size to remain anchored to the physicalsurface 16 a. For example, the user may be detected as moving backwardor forwardly, or left or right relative to the projector 4005 by thesensor 4007. The input sensor 4007 may communicate a position change bythe user and the processor 4020 may control the projected image tobecome larger or smaller, or more the projected image to a comfortableviewing location associated with the user's new location. For example,the projected image may move from a horizontal surface to a verticalsurface to follow the user.

A mobile computing device 4010, such as a smartphone may be configuredto support the gesture user interface functionality of the projector4005, which projects a computer generated image (such as a presentationslide, text document or photograph) onto a projection surface 16. Acamera (image sensor) 4007 (FIG. 40A) coupled to the mobile computingdevice 4010 may be positioned so its field of view encompasses at leastpart of the displayed image. In another embodiment, a user may also usea laser pointer to place a laser spot onto the displayed image, whichmay be detected by the camera 4007.

The camera 4007 may obtain a digital image of the laser spot and atleast part of the displayed image and provide the camera image to theprocessor 4020 of the mobile computing device 4010 with a location ofthe laser spot within the image. The mobile computing device 4010processor 4020 may be configured with software instructions to analyzethe image generated by the camera 4007 (referred to herein as the“received camera image”) to recognize at least a portion of the displaycontent that is being projected onto the projection surface 16 anddetermine the location of the laser spot with respect to the content.Since laser pointers emit a bright beam of light at a specificwavelength, the laser spot can easily be recognized by the processor4020 and distinguished from the projected image based upon the intensityand/or color elements. The camera 4007 may be further configured totrack the movement of the laser spot and correlate that movement topredefined laser gestures. When a laser gesture is recognized, theprocessor 4020 may execute the corresponding function.

FIG. 40C is a high level diagram of a projector outputting multipledifferent projected images based on one or more detected users. FIG. 40Cillustrates a projector 4005 that uses a mobile communication deviceprocessor 4010 (FIG. 40A) to detect gestures from tracked images ofadditional users that have joined the collaboration. In response todetecting the new user, the processor may render a third virtual object14 c in a modular fashion that is correctly oriented to the third user.Likewise, if an additional user joins and is detected, the processor mayrender a fourth virtual object (not shown) in a modular fashion where atleast two projected images may render the same content and images.Further, a third or fourth user may use the mobile computing device 4010to change the display of the projected images 14 a, 14 b and 14 c. Forexample, the user may close the third object 14 c and open a new objecthaving different content.

Using the projector 4005, the mobile user experience may be enhanced byproviding a large input and display environment that is highly portable.Digital collaboration environments for multiple users may be set up inan ad hoc manner by using mobile devices 4010. The physical connectors4008 on the projectors 4005 may provide relative positional awarenessbetween projectors. The position data may be provided with a high degreeof accuracy. This allows users to easily create collaborative datapresentations and manipulations using multiple projected virtual objects14 a-14 c.

FIG. 40D illustrates an embodiment method 4045 for projecting images ina convenient manner for a user based on a detected object/gesture.Method 4045 may be implemented in a computing device having a processorconfigured with processor-executable instructions to perform theoperations of the method 4045. In method 4045, the processor may captureimages of a user and may capture spatial data with a sensor array inblocks 4046 and 4047. In block 4048, the processor may recognize objectsin the image and determine distances to the objects in the image. Inblock 4049, the processor may track objects in the image and may applyan anatomical analysis to determine if there is a body part within theimage.

For example, the processor may apply an anatomical analysis to thecaptured images to determine whether the captured image includes a headand shoulders, torso, and arms etc. or any other body part that wouldindicate a user or the direction in which a user is looking. If a bodypart is detected, the processor may determine if the detected body partover time conforms to gestures stored in a memory (e.g., in a gesturedictionary) and may apply a command when a gesture is recognized. In anembodiment, the pico projector may project the image in a specificlocation so the user can conveniently watch and interact with theprojected images. For example, if a head and shoulders are determined tobe in the image, the pico projector may project an image on a wall or ona table top conveniently near the user.

In a further embodiment, users may apply or wear on one or both hands orwrists gloves, rings, bracelets, and or other items that are colored orfeature reflective features that can be recognized by one or both of thedistance sensors and cameras in order to aid in distinguishing theuser's hands and fingers and recognizing movement gestures. For example,a user may wear a glove that has different colors exhibited on each ofthe fingers, and the head mounted device may be configured to recognizeeach of the finger colors based on their respective red-green-blueintensity values within pixels of the scene image.

In blocks 4050 and 4051, the processor receives a first and a secondsignal from the respective first and second pico projector to link andcommunicate data between the pico projectors and the processor. Forexample, the pico projectors may link with a mobile communication deviceso data content of a memory associated with the mobile communicationdevice memory may be accessed and displayed or shared. In determinationblock 4052, the processor determines whether to project images from thepico projectors based on a detected body part. If so, (i.e.,determination block 4052=“Yes”), the processor may display contentutilizing the projected image at a surface convenient for a user inblock 4053. In block 4054, the processor may share data between theusers and a wireless data link between the pico projectors. For example,image files, word processing files, e-mails or any other data may beshared or displayed as a projected image. If the processor determinesnot to project the images based on the detected body part, (i.e.,determination block 4052=“No”), the processor may display a prompt toproject an image and project the image in a conventional manner inblocks 4055 and 4056. For example, the pico projector may hold offprojecting the projections until an input is received and the user maymanually detect the projected images. The projected images may include avirtual object and the head mounted devices may be configured torecognize user hand gestures interacting with or relating to theprojected virtual object image as user inputs. In such embodiments, theprocessor may apply a command corresponding to a recognized hand gesturethat is made when utilizing the projected virtual input device.

For example, a virtual mouse or virtual pointer may be displayed thatcan be moved with tracked movements of a user's hand. Additionally,using the detected gesture, the processor may correctly orient theprojected images on a vertical or horizontal surface. In this manner,the projected image may be projected right side up based on the detectedbody part. For example, the first pico projector may be connected to thesecond projector in a back to back configuration and the projectors mayproject at least two different projected images with the projectedimages being separated by at least one hundred eighty degrees relativeto one another. In yet another embodiment, the processor may define areference point and determine an orientation of each of the first andthe second projectors based on the reference point. The processor mayorient the projected image for the users based on the reference pointand the orientation of the first and the second projectors.

In another embodiment, the processor may detect a third user and projectat least a third projected image based on a detected body part. Also,the processor may receive data from a third computing device associatedwith the third user for collaboration. In another embodiment, a fourthpico projector may be provided and at least four different projectedimages.

Various embodiments may include methods of projecting images from firstand second pico projectors coupled to a computing device by capturingimages of a scene in a vicinity of the first and second projectors,capturing spatial data in the vicinity of the first and secondprojectors with a sensor array, analyzing captured scene images torecognize body parts, and projecting images from each of the first andthe second projectors with a shape and orientation determined based onthe recognized body parts. In an embodiment, the first and the secondprojectors may be pico projectors. In a further embodiment, projectingimages from each of the first and the second projectors with a shape andorientation determined based on the recognized body parts may includeprojecting images on a surface in front of users in the vicinity of theprojectors.

In a further embodiment, the method may include orienting the projectedimages on a vertical or horizontal surface appropriate for users in thevicinity of the projectors. In a further embodiment, the method mayinclude tracking movements of the recognized body parts, applying adetection algorithm to the tracked movements to detect a predeterminedgesture, applying a command corresponding to the detected predeterminedgesture, and modifying the projected images in response to the appliedcommand. In a further embodiment, analyzing captured images to recognizebody parts may include applying an anatomical analysis to the images todetermine whether objects within the captured images match a portion ofan anatomical model.

In a further embodiment, the projected images may be of a virtual inputdevice, and applying a command corresponding to the detectedpredetermined gesture may include applying a command consistent withthose of a user input device. In a further embodiment, applying acommand corresponding to the detected predetermined gesture may includeapplying a command corresponding to moving the virtual object, andmodifying the projected images in response to the applied command mayinclude displaying the virtual object as moving from a first location toa second location in response to detecting the predetermined gesture. Ina further embodiment, the method may include connecting the firstprojector to the second projector via connectors on the exterior of eachprojector in a back to back configuration, in which projecting imagesfrom each of the first and the second projectors with a shape andorientation determined based on the recognized body parts may includeprojecting at least two images in approximately opposite directions.

In a further embodiment, the method may include connecting the firstprojector to a third projector via connectors on exteriors of theprojectors, and connecting the third projector to the second projectorvia connectors on exteriors of the projectors, in which projectingimages from each of the first and the second projectors with a shape andorientation determined based on the recognized body parts may includeprojecting at least three projected images such that they are separatedby approximately ninety degrees.

In a further embodiment, the method may include connecting the firstprojector to a third projector and a fourth projector via connectors onexteriors of the projectors, and connecting the third and fourthprojectors to the second projector via connectors on exteriors of theprojectors, in which projecting images from each of the first and thesecond projectors with a shape and orientation determined based on therecognized body parts may include projecting four images such that theyare separated by approximately ninety degrees.

In a further embodiment, the method may include defining a referencepoint in the scene, determining an orientation of each of the first andthe second projectors based on the defined reference point, andorienting the projected images based on the defined reference point andthe orientation of the first and the second projectors. In a furtherembodiment, defining a reference point in the scene may includeconnecting the first projector to the second projector by a connectionpoint, and defining the connection point as the reference point.

Various embodiments may include methods of rendering virtual images inan augmented reality system by capturing an image with a body mountedcamera, capturing spatial data with a body mounted sensor array,recognizing an object within the captured image, receiving a first userinput indicating a first anchor surface in the captured image thatcorresponds to a first surface located in the image, calculatingparameters including distance and orientation with respect to the bodymounted camera that correspond to the first anchor surface, displaying avirtual object so the virtual object appears anchored to the selectedfirst anchor surface, receiving a second input indicating a secondanchor surface within the captured image that is different from thefirst anchor surface, in which the second anchor surface corresponds toa second surface located in the image and the first and second surfacesare different, calculating parameters including distance and orientationwith respect to the body mounted camera that corresponds to the secondanchor surface, and displaying the virtual object so the virtual objectappears to the user to be anchored to the selected second anchor surfaceand moved from the first anchor surface.

In an embodiment, the method may include continuously updating thedisplay of the generated virtual object so the virtual object appearsanchored to the selected first or second anchor surface as the userturns his/her head and moves with respect to the selected first orsecond anchor surface. In a further embodiment, displaying the virtualobject may include displaying a virtual head mounted display. The methodmay further include outputting video images as if appearing on the headmounted display anchored to the selected first or second anchor surface.In a further embodiment, displaying the generated virtual object so thevirtual object appears to the user to be anchored to the selected secondanchor surface and moved from the first anchor surface may includegenerating the virtual object as a rectangular shaped virtual objectthat resembles a flat screen display, and displaying the generatedvirtual object on a semitransparent display with the virtual objectsuperimposed on a real world surface visible through the semitransparentdisplay.

In a further embodiment, the first or second anchor surface may includea wall in a room, and in which the virtual object remains anchored onthe wall in the room as the user moves about in the room. In a furtherembodiment, displaying the generated virtual object so the virtualobject appears anchored to the selected first or second anchor surfacemay include generating the virtual object so that it appears to becontacting the first or second anchor surface and so that the first orsecond anchor surface appear to be a rigid point of support for theanchored virtual object. In a further embodiment, displaying thegenerated virtual object so the virtual object appears anchored to theselected anchor surface may include generating the virtual object sothat it appears to be spaced apart from the first or second anchorsurface but hovering in a fixed position and spaced relative to thefirst or second anchor surface.

In a further embodiment, the method may include transmitting data by thecaptured spatial data, the generated virtual object and the image to asecond head mounted display. In a further embodiment, the method mayinclude receiving audio from the user utilizing a microphone, applying adetection algorithm to the received audio to detect a predeterminedaudio signal, and implementing the predefined user input when thepredetermined audio signal is detected. In a further embodiment,applying a detection algorithm to the received audio to detect apredetermined audio signal may include applying a detection algorithm tothe received audio to detect a predetermined audio signal spoken by theuser corresponding to a predefined user input. In a further embodiment,the predefined user input corresponding to the detected predeterminedaudio signal is one of the first input, the second input or aconfirmation of one of the first and second inputs.

In a further embodiment, the method may include capturing the image withthe body mounted camera that is mounted and disposed together with ahead mounted display, and displaying the virtual object on the headmounted display. In a further embodiment, the body mounted camera andhead mounted display include a body mounted device. The method furthermay further include establishing a wireless data link between the bodymounted device and a second body mounted device, receiving audio datafrom the second body mounted device via the wireless data link, andusing the received audio data to output audio to the user. In a furtherembodiment, the method may include receiving video data via the wirelessdata link, and using the received video data to output video images onthe head mounted display received from the second body mounted device,in which the video displays an anchored virtual object.

In a further embodiment, receiving video data via the wireless data linkmay include receiving one or more of a three dimensional model data fora virtual object, three dimensional data for an anchoring surface, threedimensional data for an anchoring point, three dimensional depth data,and three dimensional distance data. In a further embodiment, the methodmay include receiving a third input indicating a standby location forthe virtual object, in which the standby location is different from thefirst and the second anchor surfaces, calculating parameters includingdistance and orientation with respect to the body mounted camera thatcorresponds to the standby location, and displaying the virtual objectat the standby location. In a further embodiment, the standby locationis displayed as free floating on a display, in which the virtual objectappears on the display as movable and unconnected to surfaces in theimage as the user turns his/her head and moves.

In a further embodiment, capturing an image with a body mounted cameramay include capturing video data by a full-color video camera, andcapturing spatial data with a body mounted sensor array may includecapturing spatial data utilizing a distance sensor to determinedistances to objects in the image. In a further embodiment, capturingspatial data with a body mounted sensor array may include capturingdistance information of objects in the image using an infrared laser andan infrared sensor. In a further embodiment, the method may includebuilding a three dimensional map based on the captured video data andthe calculated spatial data. In a further embodiment, the method mayinclude tracking movements of the user utilizing the body mountedcamera, applying a detection algorithm to the tracked user movements todetect a predetermined gesture, and executing a command corresponding tothe predetermined gesture when the predetermined gesture is detected.

In a further embodiment, the detected predetermined gesture is selectedfrom the group consisting of an “OK” sign, a fist, an open hand,pointing with one finger, pointing with two fingers, pointing with threeor four fingers, an outstretched hand, a hand rotation, a wave, amovement of one or more fingers, a movement of a body part or a foot,and any combination thereof. In a further embodiment, the predeterminedgesture is retrieved from a gesture dictionary. In a further embodiment,the gesture dictionary defines one or more of a poke gesture, a patgesture, or a tap gesture for target selection or toggling, a pushgesture for guiding a screen object across a screen, a flick gesture forimparting momentum to the screen object, a turn gesture for rotating thescreen object, a grab gesture or a pull gesture for zoom operations, atwo hand move gesture for panning operations, a drawing hands apartgesture for resizing the screen object, and a swipe gesture forhorizontal scrolling or shuttling through media.

In a further embodiment, receiving the first input indicating a firstanchor surface in the captured image may include detecting a firstpredetermined gesture made by the user corresponding to a command todesignate a surface within the captured image as an anchor surface andapplying the command, and receiving the second input to anchor thevirtual object in the image may include detecting a second predeterminedgesture made by the user corresponding to a command to move the anchorsurface to another surface in the captured image and applying thecommand. In a further embodiment, either of the first or second detectedpredetermined gestures may include the user pointing to a surface in thecaptured image. In a further embodiment, the method may includecontinuing tracking movements of the user after the first input isreceived, applying a detection algorithm to the tracked movements todetect a third predetermined gesture corresponding to a user inputconfirming the first input, and anchoring the virtual object to theselected anchor surface in response to the third predetermined gesture.

Various embodiments may include methods of storing spatial data for usein an augmented reality system by generating data regarding locations ofsurfaces and objects in a scene based on images and spatial datagathered from a first body mounted sensor device, generating a threedimensional map of the scene based on the generated data, addinggeographical identification metadata to the three dimensional map of thescene, storing the geographical identification metadata and threedimensional map in a memory, and transmitting at least a portion of thegeographical identification metadata and three dimensional map to asecond body mounted sensor device. In an embodiment, generating dataregarding locations of surfaces and objects in a scene based on imagesand spatial data gathered from a first body mounted sensor device mayinclude capturing a scene image of the scene with a body mounted camera,capturing spatial data of objects within the scene with a body mountedsensor array, recognizing objects within the captured scene image, anddetermining distances from the first body mounted sensor device to theobjects recognized within the scene image, and adding geographicalidentification metadata to the three dimensional map of the scene mayinclude associating the determined distances with the objects recognizedwithin the scene image.

In an embodiment, the method may include determining a location of thesecond body mounted sensor device, in which transmitting at least aportion of the geographical identification metadata and threedimensional map to the second body mounted sensor device may includetransmitting a portion of the geographical identification metadata andthree dimensional map relevant to the location of the second bodymounted sensor device. In a further embodiment, the method may includereceiving the transmitted portion of the geographical identificationmetadata and three dimensional map in the second body mounted sensordevice, and generating a visual image on a display on the second bodymounted sensor device based in part on the received portion of thegeographical identification metadata and three dimensional map.

In a further embodiment, the method may include determining a locationof the second body mounted sensor device, generating data regardinglocations of surfaces and objects in the scene based on images andspatial data gathered from the second body mounted sensor device, andrefining the geographical identification metadata and the threedimensional map based upon the data generated by the second body mountedsensor device. In a further embodiment, adding geographicalidentification metadata to the three dimensional map of the scene mayinclude tagging the three dimensional map with a geotag. In a furtherembodiment, generating data regarding locations of surfaces and objectsin a scene based on images and spatial data gathered from a first bodymounted sensor device, generating a three dimensional map of the scenebased on the generated data, and adding geographical identificationmetadata to the three dimensional map of the scene are accomplished in aprocessor within the first body mounted sensor device.

In a further embodiment, the method may include relaying images andspatial data gathered by the first body mounted sensor device to aprocessor in communication with the first body mounted sensor device, inwhich generating data regarding locations of surfaces and objects in ascene based on images and spatial data gathered from a first bodymounted sensor device, generating a three dimensional map of the scenebased on the generated data, and adding geographical identificationmetadata to the three dimensional map of the scene are accomplished inthe processor in communication with the first body mounted sensordevice. In a further embodiment, the method may include transmittingimages and spatial data gathered by the first body mounted sensor deviceto a server via a network, in which generating data regarding locationsof surfaces and objects in a scene based on images and spatial datagathered from a first body mounted sensor device, generating a threedimensional map of the scene based on the generated data, and addinggeographical identification metadata to the three dimensional map of thescene are accomplished in the server based on the image and spatial datareceived from the first body mounted sensor device.

In a further embodiment, the method may include determining a locationof the second body mounted sensor device, and transmitting a requestfrom the second body mounted sensor device to the server for a portionof the geographical identification metadata and three dimensional map,the request including the determined location of the second body mountedsensor device, in which transmitting at least a portion of thegeographical identification metadata and three dimensional map to thesecond body mounted sensor device may include transmitting from theserver to the second body mounted sensor device a portion of thegeographical identification metadata and three dimensional map that isrelevant to the location of the second body mounted sensor device. In afurther embodiment, generating a three dimensional map of the scene mayinclude developing a mathematical representation of three-dimensionalsurfaces within the scene, and writing the geographical identificationmetadata into the mathematical representation. In a further embodiment,storing the geographical identification metadata and the threedimensional map to a memory may include storing the mathematicalrepresentation on a server accessible by the first and second bodymounted sensor devices.

In a further embodiment, generating data regarding locations of surfacesand objects in a scene based on images and spatial data gathered from afirst body mounted sensor device may include determining a location ofthe first body mounted sensor device, capturing a scene image of thescene with a body mounted camera, capturing spatial data of objectswithin the scene with a body mounted sensor array, recognizing objectswithin the captured scene image, and determining locations of theobjects recognized within the scene image, and adding geographicalidentification metadata to the three dimensional map of the scene mayinclude associating the determined locations with the objects recognizedwithin the scene image. In a further embodiment, determining locationsof the objects recognized within the image may include determininglocations of the objects with respect to the determined location of thefirst body mounted sensor device. In a further embodiment, determininglocations of the objects recognized within the image may includedetermining locations of the objects within a coordinate system used indetermining the location of the first body mounted sensor device.

Various embodiments may include methods of rendering images in anaugmented reality system by capturing an image of a scene in which avirtual object is to be displayed, recognizing a body part present inthe captured image, and adjusting a display of the virtual object basedupon the recognized body part. In an embodiment, adjusting a display ofthe virtual object may include identifying a portion of the virtualobject that overlaps the recognized body part, and suppressing theidentified portion such that the recognized body part appears to occludethe identified portion of the virtual object. In a further embodiment,adjusting a display of the virtual object may include identifying aportion of the virtual object that overlaps the recognized body part,and superimposing the virtual object on the recognized body part suchthat the identified portion of the virtual object appears to occlude therecognized body part.

In a further embodiment, adjusting the displayed virtual object mayinclude adjusting displayed portions of the virtual object overlappingthe recognized body part such that the virtual object is viewablethrough the recognized body part and such that the recognized body partappears semitransparent. In a further embodiment, the method may includedisplaying the virtual object on a body mounted semitransparent displayconfigured such that that an individual wearing the body mountedsemitransparent display can view the scene through the display. In afurther embodiment, capturing an image of a scene in which a virtualobject is to be displayed may include capturing an image with a bodymounted camera worn by a first individual. The method may furtherinclude capturing spatial data with a body mounted sensor array worn bythe first individual, recognizing objects within the captured image, anddetermining distances to the recognized objects within the capturedimage. In a further embodiment, the method may include displaying thevirtual object and the captured image on a head mounted display suchthat the scene is viewable to an individual wearing the body mountedsemitransparent display.

In a further embodiment, recognizing a body part present in the capturedimage may include applying an anatomical analysis to the captured image.In a further embodiment, applying an anatomical analysis to the imagemay include recognizing objects within the captured image, comparing therecognized objects to a skeletal model stored in memory, the storedskeletal model including relative position information about key joints,bones, and limbs of a human skeleton, and determining if any of therecognized objects match a portion of the skeletal model in response tocomparing the recognized objects to the skeletal model. In a furtherembodiment, the method may include determining whether the body partbelongs to a first individual or to a second individual in response torecognizing a body part present in the captured image, determining alocation and encompassed area of the virtual object, and determiningwhether the recognized body part overlaps at least one portion of thearea encompassed by the virtual object.

In a further embodiment, determining whether the body part belongs to afirst individual or to a second individual may include determiningwhether the body part belongs to the first or second individual based onan orientation of the recognized body part. In a further embodiment,determining whether the body part belongs to a first individual or to asecond individual may include determining that the body part belongs tothe first or second individual based on a distance of the recognizedbody part from the body mounted camera. In a further embodiment,determining whether the body part belongs to a first individual or to asecond individual may include determining that the body part belongs tothe first or second individual based on a number of body partsdetermined to be in the captured image.

In a further embodiment, adjusting a display of the virtual object mayinclude displaying the virtual object superimposed over the recognizedbody part of the second individual when it is determined that the bodypart belongs to the second individual, and displaying the virtual objectso that the virtual object appears to be located beneath the recognizedbody part when it is determined that the body part belongs to the firstindividual. In a further embodiment, displaying the virtual object sothat the virtual object appears to be located beneath the recognizedbody part may include not rendering portions of the virtual object whichoverlap the recognized body part. In a further embodiment, the methodmay include displaying an outline of portions of the recognized bodypart of the second individual that overlap a display area encompassed bythe virtual object such that the outline is visible over the virtualobject in the display.

Various embodiments may include methods of rendering images in anaugmented reality system by first and second body mounted devices eachincluding a body mounted camera and a head mounted display. The methodmay include capturing a first scene image of a scene with the bodymounted camera of the first body mounted device, displaying on the headmounted display of the first body mounted device a first image of avirtual object, the first image displaying the virtual object as beinganchored to a first anchor surface within the captured first scene imagein a first orientation that is appropriate for a user of the first bodymounted device, and displaying on the head mounted display of the secondbody mounted device a second image of the virtual object, the secondimage displaying the virtual object as being anchored to a second anchorsurface in a second orientation that is appropriate for a user of thesecond body mounted device.

In a further embodiment, the method may include sending the virtualobject from the first body mounted device to the second body mounteddevice, receiving the virtual object in the second body mounted device,capturing a second scene image of the scene with the body mounted cameraof the second body mounted device, and analyzing the captured secondscene image to identify a suitable anchor surface within the secondscene image that is in a direction close to a direction that the user ofthe second body mounted device is looking, in which displaying on thehead mounted display of the second body mounted device a second imagemay include displaying the second image such that the virtual object isanchored to the identified suitable anchor surface. In an embodiment,the method may include transmitting a signal from the first body mounteddevice to the second body mounted device indicating an orientation ofthe virtual object, in which displaying the second image may includedisplaying the second image such that virtual object is oriented basedin part on the signal received from the first body mounted device.

In a further embodiment, the method may include transmitting a signalfrom the first body mounted device to the second body mounted deviceindicating a preferred orientation of the virtual object, informing theuser of the second body mounted device of the indicated preferredorientation, and receiving input from the user of the second bodymounted device accepting or rejecting the indicated preferredorientation, in which displaying on the head mounted display of thesecond body mounted device a second image of the virtual object mayinclude displaying the second image such that the virtual object isoriented based in part on the indicated preferred orientation when thereceived input accepts the indicated preferred orientation and such thatthe virtual object is oriented in a manner different from the indicatedpreferred orientation when the received input rejects the indicatedpreferred orientation.

In a further embodiment, the first anchor surface and the second anchorsurface are the same surface. In a further embodiment, the method mayinclude transmitting the virtual object from the first body mounteddevice to the second body mounted device, in which displaying on thehead mounted display of the second body mounted device a second image ofthe virtual object may include displaying the second image as beinganchored to the first anchor surface in the second orientation. In afurther embodiment, the first and second body mounted devices arelocated in different locations, the method may further includetransmitting the virtual object to each the first and second bodymounted devices via a network.

In a further embodiment, the method may include capturing a second sceneimage with the body mounted camera of the second body mounted device,and receiving input on the second body mounted device indicating thesecond anchor surface in the captured second scene image, the indicatedsecond anchor surface being different from the first anchor surface, inwhich displaying on the head mounted display of the second body mounteddevice a second image may include displaying the virtual object as beinganchored to the indicated second anchor surface in a configurationappropriate for the indicated second anchor surface and the user of thesecond body mounted device. In a further embodiment, the method mayinclude transmitting a second virtual object from the second bodymounted device to the first body mounted device, and displaying a thirdimage on the head mounted display of the first body mounted device, thethird image displaying the second virtual object as being anchored tothe first anchor surface.

Further embodiments may include systems, servers, and/or computingdevices configured to perform various operations corresponding to thevarious embodiment methods and/or operations discussed above. Evenfurther embodiments may include non-transitory processor-readablestorage mediums having stored thereon processor-executable instructionsto cause a processor to perform operations corresponding to the variousembodiment methods and/or operations discussed above.

FIG. 41 is a system block diagram of a receiver device suitable for usewith any of the embodiments. A typical receiver device 4100 may includea processor 4101 coupled to internal memory 4102, a display 4103, and toa speaker 4154. Additionally, the receiver device 4100 may include anantenna 4104 for sending and receiving electromagnetic radiation thatmay be connected to a wireless data link and/or cellular telephonetransceiver 4105 coupled to the processor 4101 and a mobile multimediabroadcast receiver 4106 coupled to the processor 4101. Receiver devices4100 typically also include menu selection buttons or rocker switches4108 for receiving user inputs.

The various embodiments may be implemented on any of a variety ofcommercially available server devices, such as the server 4200illustrated in FIG. 42. Such a server 4200 typically may include aprocessor 4201 coupled to volatile memory 4202 and a large capacitynonvolatile memory, such as a disk drive 4203. The server 4200 may alsoinclude a floppy disc drive, compact disc (CD) or DVD disc drive 4206coupled to the processor 4201. The server 4200 may also include networkaccess ports 4204 coupled to the processor 4201 for establishing dataconnections with a network 4205, such as a local area network coupled toother broadcast system computers and servers.

The processors 3903, 3905, 4101, 4201 may be any programmablemicroprocessor, microcomputer or multiple processor chip or chips thatcan be configured by software instructions (applications) to perform avariety of functions, including the functions of the various embodimentsdescribed above. In some devices, multiple processors 3903, 3905, 4101may be provided, such as one processor dedicated to wirelesscommunication functions and one processor dedicated to running otherapplications. Typically, software applications may be stored in theinternal memory 4102, 4202, 4203 before they are accessed and loadedinto the processor 4101, 4201. The processor 4101, 4201 may includeinternal memory sufficient to store the application softwareinstructions. In many devices the internal memory may be a volatile ornonvolatile memory, such as flash memory, or a mixture of both. For thepurposes of this description, a general reference to memory refers tomemory accessible by the processor 3903, 3905, 4101, 4201 includinginternal memory or removable memory plugged into the device and memorywithin the processor 3903, 3905, 4101, 4201 itself.

The foregoing method descriptions and the process flow diagrams areprovided merely as illustrative examples and are not intended to requireor imply that the steps of the various embodiments must be performed inthe order presented. As will be appreciated by one of skill in the artthe order of steps in the foregoing embodiments may be performed in anyorder. Words such as “thereafter,” “then,” “next,” etc. are not intendedto limit the order of the steps; these words are simply used to guidethe reader through the description of the methods. Further, anyreference to claim elements in the singular, for example, using thearticles “a,” “an” or “the” is not to be construed as limiting theelement to the singular.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described in connection with the embodiments disclosedherein may be implemented as electronic hardware, computer software, orcombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the presentinvention.

The hardware used to implement the various illustrative logics, logicalblocks, modules, and circuits described in connection with theembodiments disclosed herein may be implemented or performed with ageneral purpose processor, a digital signal processor (DSP), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA) or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. Ageneral-purpose processor may be a microprocessor, but, in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Alternatively, some steps or methods may be performed bycircuitry that is specific to a given function.

In one or more exemplary embodiments, the functions described may beimplemented in hardware, software, firmware, or any combination thereof.If implemented in software, the functions may be stored on as one ormore instructions or code on a non-transitory computer-readable medium.The steps of a method or algorithm disclosed herein may be embodied in aprocessor-executable software module executed which may reside on anon-transitory computer-readable medium. Non-transitorycomputer-readable media include any storage media may be accessed by acomputer. By way of example, and not limitation, such non-transitorycomputer-readable media may include RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that may be used to carry or store desiredprogram code in the form of instructions or data structures and that maybe accessed by a computer. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofnon-transitory computer-readable media. Additionally, the operations ofa method or algorithm may reside as one or any combination or set ofcodes and/or instructions on a non-transitory machine readable mediumand/or non-transitory computer-readable medium, which may beincorporated into a computer program product.

The preceding description of the disclosed embodiments is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these embodiments will be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other embodiments without departing from thespirit or scope of the invention. Thus, the present invention is notintended to be limited to the embodiments shown herein but is to beaccorded the widest scope consistent with the following claims and theprinciples and novel features disclosed herein.

1. A method of storing spatial data for use in an augmented reality system, comprising: generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in a memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map to a second body mounted sensor device.
 2. The method of claim 1, wherein: generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device comprises: capturing a scene image of the scene with a body mounted camera; capturing spatial data of objects within the scene with a body mounted sensor array; recognizing objects within the captured scene image; and determining distances from the first body mounted sensor device to the objects recognized within the scene image; and adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined distances with the objects recognized within the scene image.
 3. The method of claim 1, further comprising determining a location of the second body mounted sensor device, wherein: transmitting at least a portion of the geographical identification metadata and three dimensional map to the second body mounted sensor device comprises transmitting a portion of the geographical identification metadata and three dimensional map relevant to the location of the second body mounted sensor device.
 4. The method of claim 1, further comprising: receiving the transmitted portion of the geographical identification metadata and three dimensional map in the second body mounted sensor device; and generating a visual image on a display on the second body mounted sensor device based in part on the received portion of the geographical identification metadata and three dimensional map.
 5. The method of claim 1, further comprising: determining a location of the second body mounted sensor device; generating data regarding locations of surfaces and objects in the scene based on images and spatial data gathered from the second body mounted sensor device; and refining the geographical identification metadata and the three dimensional map based upon the data generated by the second body mounted sensor device.
 6. The method of claim 1, wherein adding geographical identification metadata to the three dimensional map of the scene comprises tagging the three dimensional map with a geotag.
 7. The method of claim 1, wherein generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device, generating a three dimensional map of the scene based on the generated data, and adding geographical identification metadata to the three dimensional map of the scene are accomplished in a processor within the first body mounted sensor device.
 8. The method of claim 1, further comprising: relaying images and spatial data gathered by the first body mounted sensor device to a processor in communication with the first body mounted sensor device, wherein generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device, generating a three dimensional map of the scene based on the generated data, and adding geographical identification metadata to the three dimensional map of the scene are accomplished in the processor in communication with the first body mounted sensor device.
 9. The method of claim 1, further comprising: transmitting images and spatial data gathered by the first body mounted sensor device to a server via a network, wherein generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device, generating a three dimensional map of the scene based on the generated data, and adding geographical identification metadata to the three dimensional map of the scene are accomplished in the server based on the image and spatial data received from the first body mounted sensor device.
 10. The method of claim 9, further comprising: determining a location of the second body mounted sensor device; and transmitting a request from the second body mounted sensor device to the server for a portion of the geographical identification metadata and three dimensional map, the request including the determined location of the second body mounted sensor device, wherein transmitting at least a portion of the geographical identification metadata and three dimensional map to the second body mounted sensor device comprises transmitting from the server to the second body mounted sensor device a portion of the geographical identification metadata and three dimensional map that is relevant to the location of the second body mounted sensor device.
 11. The method of claim 1, wherein generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 12. The method of claim 11, wherein storing the geographical identification metadata and the three dimensional map to a memory comprises: storing the mathematical representation on a server accessible by the first and second body mounted sensor devices.
 13. The method of claim 1, wherein: generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device comprises: determining a location of the first body mounted sensor device; capturing a scene image of the scene with a body mounted camera; capturing spatial data of objects within the scene with a body mounted sensor array; recognizing objects within the captured scene image; and determining locations of the objects recognized within the scene image; and adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined locations with the objects recognized within the scene image.
 14. The method of claim 13, wherein determining locations of the objects recognized within the image comprises determining locations of the objects with respect to the determined location of the first body mounted sensor device.
 15. The method of claim 13, wherein determining locations of the objects recognized within the image comprises determining locations of the objects within a coordinate system used in determining the location of the first body mounted sensor device.
 16. A system, comprising: a first body mounted sensor device comprising a first memory, first transceiver and a first device processor coupled to the first transceiver and first memory; and a second body mounted sensor device comprising a display, a second transceiver and a second device processor coupled to the display and second transceiver, wherein the first device processor is configured with processor-executable instructions to perform operations comprising: gathering images and spatial data; generating data regarding locations of surfaces and objects in a scene based on the gathered images and spatial data; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in the first memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map to the second body mounted sensor device, and wherein the second device processor is configured with processor-executable instructions to perform operations comprising: receiving the transmitted portion of the geographical identification metadata and three dimensional map; and generating a visual image on the display based in part on the received portion of the geographical identification metadata and three dimensional map.
 17. The system of claim 16, wherein the first device processor is configured with processor-executable instructions to perform operations such that: generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device comprises: capturing a scene image of the scene with a body mounted camera; capturing spatial data of objects within the scene with a body mounted sensor array; recognizing objects within the captured scene image; and determining distances from the first body mounted sensor device to the objects recognized within the scene image; and adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined distances with the objects recognized within the scene image.
 18. The system of claim 16, wherein: the first device processor is configured with processor-executable instructions to perform operations further comprising determining a location of the second body mounted sensor device; and the first device processor is configured with processor-executable instructions to perform operations such that transmitting at least a portion of the geographical identification metadata and three dimensional map to the second body mounted sensor device comprises transmitting a portion of the geographical identification metadata and three dimensional map relevant to the location of the second body mounted sensor device.
 19. The system of claim 16, wherein the first device processor is configured with processor-executable instructions to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises tagging the three dimensional map with a geotag.
 20. The system of claim 16, wherein the first device processor is configured with processor-executable instructions to perform operations such that generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 21. The system of claim 20, wherein the first device processor is configured with processor-executable instructions to perform operations such that storing the geographical identification metadata and the three dimensional map to a memory comprises: storing the mathematical representation on a server accessible by the first and second body mounted sensor devices.
 22. The system of claim 16, wherein the first device processor is configured with processor-executable instructions to perform operations such that: generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a first body mounted sensor device comprises: determining a location of the first body mounted sensor device; capturing a scene image of the scene with a body mounted camera; capturing spatial data of objects within the scene with a body mounted sensor array; recognizing objects within the captured scene image; and determining locations of the objects recognized within the scene image; and adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined locations with the objects recognized within the scene image.
 23. The system of claim 22, wherein the first device processor is configured with processor-executable instructions to perform operations such that determining locations of the objects recognized within the image comprises determining locations of the objects with respect to the determined location of the first body mounted sensor device.
 24. The system of claim 22, wherein the first device processor is configured with processor-executable instructions to perform operations such that determining locations of the objects recognized within the image comprises determining locations of the objects within a coordinate system used in determining the location of the first body mounted sensor device.
 25. The system of claim 16, wherein the second device processor is configured with processor-executable instructions to perform operations further comprising: gathering images and spatial data; determining a location of the second body mounted sensor device; generating data regarding locations of surfaces and objects in the scene based on the gathered images and spatial data; and refining the geographical identification metadata and the three dimensional map based upon the generated data.
 26. A system, comprising: a first body mounted sensor device comprising a first memory, a first transceiver and a first device processor coupled to the first transceiver and first memory; a second body mounted sensor device comprising a display, a second transceiver and a second device processor coupled to the display and second transceiver; and a server comprising a server memory and a server processor coupled to the server memory, wherein the first device processor is configured with processor-executable instructions to perform operations comprising: gathering images and spatial data; transmitting the gathered images and spatial data to the server, wherein the server processor is configured with server-executable instructions to perform operations comprising: receiving the gathered images and spatial data from the first body mounted sensor device; generating data regarding locations of surfaces and objects in a scene based on the received images and spatial data; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in the server memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map to the second body mounted sensor device, and wherein the second device processor is configured with processor-executable instructions to perform operations comprising: receiving the transmitted portion of the geographical identification metadata and three dimensional map; and generating a visual image on the display based in part on the received portion of the geographical identification metadata and three dimensional map.
 27. The system of claim 26, wherein the second device processor is configured with processor-executable instructions to perform operations further comprising: determining a location of the second body mounted sensor device; and transmitting a request to the server for a portion of the geographical identification metadata and three dimensional map, the request including the determined location of the second body mounted sensor device.
 28. The system of claim 27, wherein: the server processor is configured with server-executable instructions to perform operations further comprising receiving the location of the second body mounted sensor device transmitted by the second body mounted sensor device; and the server processor is configured with server-executable instructions to perform operations such that transmitting at least a portion of the geographical identification metadata and three dimensional map comprises transmitting a portion of the geographical identification metadata and three dimensional map that is relevant to the location of the second body mounted sensor device.
 29. The system of claim 26, wherein the server processor is configured with server-executable instructions to perform operations such that generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 30. The system of claim 29, wherein the server processor is configured with server-executable instructions to perform operations such that storing the geographical identification metadata and the three dimensional map in a server memory comprises storing the mathematical representation such that it is accessible by the first and second body mounted sensor devices.
 31. The system of claim 26, wherein the first body mounted sensor device further comprises a body mounted camera and a body mounted sensor array, and the first device processor is configured with processor-executable instructions to perform operations such that gathering images and spatial data comprises: capturing a scene image of the scene with the body mounted camera; and capturing spatial data of objects within the scene with a body mounted sensor array.
 32. A server of an augmented reality system, comprising a memory; and a processor coupled to the memory, wherein the processor is configured with processor-executable instructions to perform operations comprising: receiving images and spatial data from a first body mounted sensor device; generating data regarding locations of surfaces and objects in a scene based on the received images and spatial data; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in the memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map to a second body mounted sensor device.
 33. The server of claim 32, wherein: the processor is configured with processor-executable instructions to perform operations further comprising receiving a location of the second body mounted sensor device; and the processor is configured with processor-executable instructions to perform operations such that transmitting at least a portion of the geographical identification metadata and three dimensional map comprises transmitting a portion of the geographical identification metadata and three dimensional map that is relevant to the location of the second body mounted sensor device.
 34. The server of claim 32, wherein the processor is configured with processor-executable instructions to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises tagging the three dimensional map with a geotag
 35. The server of claim 32, wherein the processor is configured with processor-executable instructions to perform operations such that generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 36. The server of claim 35, wherein the processor is configured with processor-executable instructions to perform operations such that storing the geographical identification metadata and the three dimensional map in the memory comprises storing the mathematical representation such that it is accessible by the first and second body mounted sensor devices.
 37. A body mounted sensor device, comprising: means for generating data regarding locations of surfaces and objects in a scene based on images and spatial data; means for generating a three dimensional map of the scene based on the generated data; means for adding geographical identification metadata to the three dimensional map of the scene; means for storing the geographical identification metadata and three dimensional map in a memory; and means for transmitting at least a portion of the geographical identification metadata and three dimensional map.
 38. The body mounted sensor device of claim 37, wherein: means for generating data regarding locations of surfaces and objects in a scene based on images and spatial data comprises: means for capturing a scene image of the scene with a body mounted camera; means for capturing spatial data of objects within the scene with a body mounted sensor array; means for recognizing objects within the captured scene image; and means for determining distances from the body mounted sensor device to the objects recognized within the scene image; and means for adding geographical identification metadata to the three dimensional map of the scene comprises means for associating the determined distances with the objects recognized within the scene image.
 39. The body mounted sensor device of claim 37, further comprising means for determining a location of a body mounted sensor device, wherein means for transmitting at least a portion of the geographical identification metadata and three dimensional map comprises means for transmitting a portion of the geographical identification metadata and three dimensional map relevant to the location of the body mounted sensor device.
 40. The body mounted sensor device of claim 37, wherein means for adding geographical identification metadata to the three dimensional map of the scene comprises means for tagging the three dimensional map with a geotag.
 41. The body mounted sensor device of claim 37, wherein means for generating a three dimensional map of the scene comprises: means for developing a mathematical representation of three-dimensional surfaces within the scene; and means for writing the geographical identification metadata into the mathematical representation.
 42. The body mounted sensor device of claim 41, wherein means for storing the geographical identification metadata and the three dimensional map to a memory comprises means for storing the mathematical representation on a server.
 43. The body mounted sensor device of claim 37, wherein: means for generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a body mounted sensor device comprises: means for determining a location of the body mounted sensor device; means for capturing a scene image of the scene with a body mounted camera; means for capturing spatial data of objects within the scene with a body mounted sensor array; means for recognizing objects within the captured scene image; and means for determining locations of the objects recognized within the scene image; and means for adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined locations with the objects recognized within the scene image.
 44. The body mounted sensor device of claim 43, wherein means for determining locations of the objects recognized within the image comprises means for determining locations of the objects with respect to the determined location of the body mounted sensor device.
 45. The body mounted sensor device of claim 43, wherein means for determining locations of the objects recognized within the image comprises means for determining locations of the objects within a coordinate system used in determining the location of the body mounted sensor device.
 46. A body mounted sensor device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured with processor-executable instructions to perform operations comprising: generating data regarding locations of surfaces and objects in a scene based on images and spatial data; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in the memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map.
 47. The body mounted sensor device of claim 46, further comprising: a body mounted camera; and a body mounted sensor array, wherein the processor is configured with processor-executable instructions to perform operations such that generating data regarding locations of surfaces and objects in a scene based on images and spatial data comprises: capturing a scene image of the scene with the body mounted camera; capturing spatial data of objects within the scene with the body mounted sensor array; recognizing objects within the captured scene image; and determining distances from the first body mounted sensor device to the objects recognized within the scene image; and wherein the processor is further configured with processor-executable instructions to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined distances with the objects recognized within the scene image.
 48. The body mounted sensor device of claim 46, wherein: the processor is configured with processor-executable instructions to perform operations further comprising determining a location of a body mounted sensor device; and the processor is configured with processor-executable instructions to perform operations such that transmitting at least a portion of the geographical identification metadata and three dimensional map comprises transmitting a portion of the geographical identification metadata and three dimensional map relevant to the location of the body mounted sensor device.
 49. The body mounted sensor device of claim 46, wherein the processor is configured with processor-executable instructions to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises tagging the three dimensional map with a geotag.
 50. The body mounted sensor device of claim 46, wherein the processor is configured with processor-executable instructions to perform operations such that generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 51. The body mounted sensor device of claim 50, wherein the processor is configured with processor-executable instructions to perform operations such that storing the geographical identification metadata and the three dimensional map to a memory comprises storing the mathematical representation on a server.
 52. The body mounted sensor device of claim 46, further comprising: a body mounted camera; and a body mounted sensor array, wherein the processor is configured with processor-executable instructions to perform operations such that generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a body mounted sensor device comprises: determining a location of the body mounted sensor device; capturing a scene image of the scene with the body mounted camera; capturing spatial data of objects within the scene with the body mounted sensor array; recognizing objects within the captured scene image; and determining locations of the objects recognized within the scene image, and wherein the processor is further configured with processor-executable instructions to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined locations with the objects recognized within the scene image.
 53. The body mounted sensor device of claim 52, wherein the processor is configured with processor-executable instructions to perform operations such that determining locations of the objects recognized within the image comprises determining locations of the objects with respect to the determined location of the body mounted sensor device.
 54. The body mounted sensor device of claim 52, wherein the processor is configured with processor-executable instructions to perform operations such that determining locations of the objects recognized within the image comprises determining locations of the objects within a coordinate system used in determining the location of the body mounted sensor device.
 55. A non-transitory computer readable storage medium having stored thereon processor-executable software instructions configured to cause a processor to perform operations comprising: generating data regarding locations of surfaces and objects in a scene based on images and spatial data; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in the memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map.
 56. The non-transitory computer readable storage medium of claim 55, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that generating data regarding locations of surfaces and objects in a scene based on images and spatial data comprises: capturing a scene image of the scene with a body mounted camera; capturing spatial data of objects within the scene with a body mounted sensor array; recognizing objects within the captured scene image; and determining distances from a body mounted sensor device to the objects recognized within the scene image, and wherein the stored processor-executable software instructions are further configured to cause a processor to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined distances with the objects recognized within the scene image.
 57. The non-transitory computer readable storage medium of claim 55, wherein: the stored processor-executable software instructions are configured to cause a processor to perform operations comprising determining a location of a body mounted sensor device; and the stored processor-executable software instructions are configured to cause a processor to perform operations such that transmitting at least a portion of the geographical identification metadata and three dimensional map comprises transmitting a portion of the geographical identification metadata and three dimensional map relevant to the location of the body mounted sensor device.
 58. The non-transitory computer readable storage medium of claim 55, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises tagging the three dimensional map with a geotag.
 59. The non-transitory computer readable storage medium of claim 55, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 60. The non-transitory computer readable storage medium of claim 59, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that storing the geographical identification metadata and the three dimensional map to a memory comprises storing the mathematical representation on a server.
 61. The non-transitory computer readable storage medium of claim 55, wherein: the stored processor-executable software instructions are configured to cause a processor to perform operations such that generating data regarding locations of surfaces and objects in a scene based on images and spatial data gathered from a body mounted sensor device comprises: determining a location of the body mounted sensor device; capturing a scene image of the scene with a body mounted camera; capturing spatial data of objects within the scene with a body mounted sensor array; recognizing objects within the captured scene image; and determining locations of the objects recognized within the scene image, and the stored processor-executable software instructions are configured to cause a processor to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises associating the determined locations with the objects recognized within the scene image.
 62. The non-transitory computer readable storage medium of claim 61, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that determining locations of the objects recognized within the image comprises determining locations of the objects with respect to the determined location of the body mounted sensor device.
 63. The non-transitory computer readable storage medium of claim 61, wherein the stored processor-executable software instructions are configured to cause a processor to perform operations such that determining locations of the objects recognized within the image comprises determining locations of the objects within a coordinate system used in determining the location of the body mounted sensor device.
 64. A non-transitory server readable storage medium having stored thereon server-executable software instructions configured to cause a server-processor to perform operations comprising: receiving images and spatial data from a first body mounted sensor device; generating data regarding locations of surfaces and objects in a scene based on the received images and spatial data; generating a three dimensional map of the scene based on the generated data; adding geographical identification metadata to the three dimensional map of the scene; storing the geographical identification metadata and three dimensional map in the memory; and transmitting at least a portion of the geographical identification metadata and three dimensional map to a second body mounted sensor device.
 65. The non-transitory server readable storage medium of claim 64, wherein: the stored server-executable instructions software instructions are configured to cause a server-processor to perform operations further comprising receiving a location of the second body mounted sensor device; and the stored server-executable instructions software instructions are configured to cause a server-processor to perform operations such that transmitting at least a portion of the geographical identification metadata and three dimensional map comprises transmitting a portion of the geographical identification metadata and three dimensional map that is relevant to the location of the second body mounted sensor device.
 66. The non-transitory server readable storage medium of claim 64, wherein the stored server-executable instructions software instructions are configured to cause a server-processor to perform operations such that adding geographical identification metadata to the three dimensional map of the scene comprises tagging the three dimensional map with a geotag.
 67. The non-transitory server readable storage medium of claim 64, wherein the stored server-executable instructions software instructions are configured to cause a server-processor to perform operations such that generating a three dimensional map of the scene comprises: developing a mathematical representation of three-dimensional surfaces within the scene; and writing the geographical identification metadata into the mathematical representation.
 68. The non-transitory server readable storage medium of claim 67, wherein the stored server-executable instructions software instructions are configured to cause a server-processor to perform operations storing the geographical identification metadata and the three dimensional map in the memory comprises storing the mathematical representation such that it is accessible by the first and second body mounted sensor devices. 